Apple Foundation Models: Schema

In terms of the Apple Foundation Models (AFM), what is a schema?

A schema is an example data structure, written in JSON format, demonstrating to the AFM, the manner in which you wish the data generated by the AFM, returned from your query.

For example, if your script is requesting the AFM to generate a list of book titles, it would include a basic Array Schema (in JSON, of course) with the query to the AFM.

Apple supports a variety of schema types and parameters to make it easier to quickly access the returned data. Those supported by Omni Automation are described here.

LanguageModel.Schema Class

This class contains the tools for creating and applying a variety of schema — from basic arrays to complex recursive objects.

There are several schema elements or nodes you can use in constructing a compatible schema for use with the AFM:

arrayOf • {type: "string representation of data type"} • Indicates that the schema is expecting an array of objects conforming to a specified data type. You can also specify the minimum and maximum number of elements the array should contain.
properties • [{name: "value"},{name: "value", isOptional: true},…] • A set of named properties used to indicate that an item has one or more identifying characteristics, such as: a “title”, a “description”, and a “priority”. Each of these properties has a required name which is how you’ll look up that property in the result. You can also optionally specify a description (so the language model has more information about the property), whether the property is optional, and a substep schema (if you want the property’s value to be something other than a simple string), and you can indicate whether the property isOptional.
anyOf • [{constant: "contant value"}, …] • Is used when the value for a parameter is to one of a set of constants.
constant • [{constant: "contant value"}, …] • A constant is a fixed value that does not change, like: "high priority" or "low priority"
referenceTo • {referenceTo: "schema name"} • Is used to reference a previously named schema to be followed to add more data to the response structure. LIke adding rows to the rows of an outline.

The following documentation contains examples of the use of each of these.

Basic Array Schema

If you wish for data returned by the AFM to be provided in simple arrays of Strings or Numbers, include one of the following basic schema with the request for data.

Apple’s basic Array Schema includes one required parameter (type) and two optional parameters (minimum and maximum):

arrayOf {"type": "String representation of data type" } • The top-level schema indicates that it’s expecting an array of objects conforming to a specified substep schema (“step”). You can also specify the minimum and maximum number of elements for the array.

Here are each of the three supported array types with and without the optional minimum and maximum parameters:

^*Array of Strings (Unlimited Response)


schema = LanguageModel.Schema.fromJSON({
	arrayOf: {type: "string"}
})

^*Array of Strings (Ranged Response)


schema = LanguageModel.Schema.fromJSON({
	arrayOf: {type: "string", minimum: 0, maximum: 10}
})

Array of Integers (Unlimited Response)


schema = LanguageModel.Schema.fromJSON({
	arrayOf: {type: "integer"}
})

Array of Integers (Ranged Response)


schema = LanguageModel.Schema.fromJSON({
	arrayOf: {type: "integer", minimum: 0, maximum: 10}
})

Array of Decimals (Unlimited Response)


schema = LanguageModel.Schema.fromJSON({
	arrayOf: {type: "decimal"}
})

Array of Decimals (Ranged Response)


schema = LanguageModel.Schema.fromJSON({
	arrayOf: {type: "decimal", minimum: -9.5, maximum: 9.5}
})

AFM Query with Schema

Apple provides two functions for making a query to the AFM: one for returning a human-readable text response, and one for returning structured data (schema).

To include the schema with the request to the AFM, use the respondWithSchema method of the LanguageModel.Session class:

respond(prompt: String) → (Promise of String) • Produces a text response to a prompt.
respondWithSchema(prompt: String, schema: LanguageModel.Schema, generationOptions: LanguageModel.GenerationOptions or null) → (Promise of String) • Produces a JSON response to a prompt using the provided schema.

Here’s a template for querying the AFM providing an array schema:

^*AFM Query with String Array Schema


(async () => {
	try {
		prompt = "Your natural language request for data"
		schema = LanguageModel.Schema.fromJSON({
			arrayOf: {type: "string"}
		})
		session = new LanguageModel.Session()
		responseJSON = await session.respondWithSchema(prompt, schema)
		response = JSON.parse(responseJSON)
		console.log(response)
	}
	catch(err){
		new Alert(err.name, err.message).show()
	}
})();

^* IMPORTANT: As of this writing, the String Array schema, used by itself (shown above example, and below “Names of the Planets”) does not produce the expected array of strings. Use the “Result Property” example schema instead.

And here’s an example of using the basic integer array schema:

AFM Query with Integer Array Schema
  

(async () => {
    try {
        prompt = "Provide a list of integers between 1 and 101 that are divisible by 4."
        schema = LanguageModel.Schema.fromJSON({
            arrayOf: {type: "integer"}
        })
        session = new LanguageModel.Session()
        responseJSON = await session.respondWithSchema(prompt, schema)
        response = JSON.parse(responseJSON)
        console.log(response)
        //--> [4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, 96, 100]
    }
    catch(err){
        new Alert(err.name, err.message).show()
    }
})();

And continuing with the “Names of the Planets” example from the first section of this documentation, here’s the basic string array:

^*Names of the Planets
  

(async () => {
    try {
        prompt = "Provide a list of the names of the planets in our solar system."
        schema = LanguageModel.Schema.fromJSON({
            arrayOf: {type: "string"}
        })
        session = new LanguageModel.Session()
        responseJSON = await session.respondWithSchema(prompt, schema)
        response = JSON.parse(responseJSON)
        console.log(response)
        //--> ["Mercury","Venus","Earth","Mars","Jupiter","Saturn","Uranus","Neptune"]
    }
    catch(err){
        new Alert(err.name, err.message).show()
    }
})();

Plug-In Template

The “AFM • Prompt for Simple Array” plug-in is a template demonstrating how to prompt for and return a simple array of data (string, integer, decimal)

Automation Console window showing the result

Object Schema with Properties

The AFM (Apple Foundation Models) is fully capable of producing responses with detail and complexity that range from simple arrays to complex outlines and discussions. Therefore, your schema must reflect the level and order of data that you wish returned. A common supported schema is requesting an array of JSON objects with properties.

To construct such schema, use the following schema elements in addition to the arrayOf element you encountered previously with the basic array schema.

name • A name this dynamic schema can be referenced by. name can also used with individual elements.
description • A natural language description of this schema or element.
properties • The properties to associated with this schema. Property names are user-defined.

For example, if we wanted an object schema for our “Names of the Planets” example, it would look something like this:

Object Schema with Properties


{
	name: "planets-schema",
	arrayOf: {
		name: "planet",
		properties: [
			{
				name: "title"
			}
		]
	}
}

Here’s the schema in use:

Query the AFM using Schema
  

(async () => {
	try {
		json = {name:"planets-schema", arrayOf:{name:"planet", properties:[{name:"title"}]}}
		schema = LanguageModel.Schema.fromJSON(json)
		prompt = "List the names of the planets in our solar system."
		session = new LanguageModel.Session()
		responseStr = await session.respondWithSchema(prompt, schema)
		console.log(responseStr)
		responseJSON = JSON.parse(responseStr)
		//--> [{"title": "Mercury"}, {"title": "Venus"}, {"title": "Earth"}, {"title": "Mars"}, {"title": "Jupiter"}, {"title": "Saturn"}, {"title": "Uranus"}, {"title": "Neptune"}]
		planetNames = responseJSON.map(item => item.title)
		console.log(planetNames)
		//--> ["Mercury", "Venus", "Earth", "Mars", "Jupiter", "Saturn", "Uranus", "Neptune"]
	}
	catch(err){
		new Alert(err.name, err.message).show()
	}
})();

Note that the parsed JSON returned by the query (line 10) is an array of JSON objects with a single property: title. To extract an array of planets names, a map(…) function is used (line 11).

The “result” Property

There is also a commonly used way to structure your schema to provide a “cleaned” result, by organizing the schema to include a response array like this:

String Array Schema with “result” Property


{
    name: "string-array-schema",
    properties: [
        {
            name: "result",
            schema: {
                arrayOf: {
                    type: "string"
                }
            }
        }
    ]
}

In use, the resulting data can be easily returned as the value of the result property (line 9):

Including “result” Property
  

(async () => {
	try {
		resultSchema = {name:"item",properties:[{name:"result",schema:{arrayOf:{type:"string"}}}]}
		schema = LanguageModel.Schema.fromJSON(resultSchema)
		prompt = 'List the names of the planets in the Solar System.'
		session = new LanguageModel.Session()
		responseStr = await session.respondWithSchema(prompt, schema);
		response = JSON.parse(responseStr)
		console.log(response.result)
	}
	catch(err){
		new Alert(err.name, err.message).show()
	}
})();

Object Schema with Multiple Properties

Staying with our “Planets of the Solar System” example, we can request more data about each planet, aside from its name. The schema has to reflect the type and organization of the data we wish returned by the AFM.

The following schema includes properties for name (title), general description, distance from the Sun, and planet composition type (as one of a list of three constants):

Object Schema with Multiple Properties (SHOW)


{ 
	name: "planets-schema", 
	description: "Information about the planets in the Solar System, including their name, general description, composition type, and distance from the Sun in astronomical units.", 
	arrayOf: { 
		name: "planet", 
		properties: [ 
			{ 
				name: "title" 
			}, 
			{ 
				name: "description"
			}, 
			{ 
				name: "distance" 
			}, 
			{ 
				name: "type", 
				schema: { 
					name: "type-schema", 
					anyOf: [ 
						{ 
							constant: "Terrestrial" 
						}, 
						{ 
							constant: "Gas Giant" 
						}, 
						{ 
							constant: "Ice Giant" 
						} 
					] 
				} 
			} 
		] 
	} 
}

Here’s a script using the condensed version of the schema to query the AFM:

Properties of the Planets
  

(async () => {
    try {
        json = {name:"planets-schema",description:"Information about the planets in the Solar System, including their name, general description, composition type, and distance from the Sun in astronomical units.",arrayOf:{name:"planet",properties:[{name:"title"},{name:"description"},{name:"distance"},{name:"type",schema:{name:"type-schema",anyOf:[{constant:"Terrestrial"},{constant:"Gas Giant"},{constant:"Ice Giant"}]}}]}}
        schema = LanguageModel.Schema.fromJSON(json)
        prompt = "Provide information about the planets in the Solar System, including their name, general description, composition type, and distance from the Sun in astronomical units."
        session = new LanguageModel.Session()
        responseStr = await session.respondWithSchema(prompt, schema)
        responseJSON = JSON.parse(responseStr)
        console.log(JSON.stringify(responseJSON, null, 2))
    }
    catch(err){
        new Alert(err.name, err.message).show()
    }
})();

And here’s a typical response. NOTE: every response may vary and contain slightly different verbiage and content. In addition , the properties may vary in their ordering within each object.

Resulting JSON


[
	{
		"type": "Terrestrial",
		"title": "Mercury",
		"description": "Mercury is the smallest and closest planet to the Sun. It has a rocky surface with craters and is very close to the Sun, resulting in extreme temperature variations.",
		"distance": "0.39 astronomical units"
	},
	{
		"description": "Venus is the second planet from the Sun and is similar in size to Earth. It has a thick, toxic atmosphere and a surface temperature hot enough to melt lead.",
		"distance": "0.72 astronomical units",
		"title": "Venus",
		"type": "Terrestrial"
	},
	{
		"type": "Terrestrial",
		"description": "Earth is the third planet from the Sun and the only known planet to support life. It has a diverse climate and abundant water resources.",
		"distance": "1 astronomical unit",
		"title": "Earth"
	},
	{
		"distance": "1.52 astronomical units",
		"type": "Terrestrial",
		"title": "Mars",
		"description": "Mars, the fourth planet from the Sun, is known as the Red Planet due to its iron oxide-rich surface. It has the largest volcano and canyon in the Solar System."
	},
	{
		"title": "Jupiter",
		"description": "Jupiter is the largest planet in the Solar System, a gas giant with a Great Red Spot, a massive storm larger than Earth. It has a strong magnetic field and many moons.",
		"type": "Gas Giant",
		"distance": "5.20 astronomical units"
	},
	{
		"distance": "9.58 astronomical units",
		"type": "Gas Giant",
		"title": "Saturn",
		"description": "Saturn is a gas giant famous for its prominent ring system. It has a composition primarily of hydrogen and helium, with a relatively small rocky core."
	},
	{
		"type": "Ice Giant",
		"title": "Uranus",
		"distance": "19.22 astronomical units",
		"description": "Uranus is an ice giant with a unique tilt, causing extreme seasonal variations. It has a blue-green color due to methane in its atmosphere."
	},
	{
		"type": "Ice Giant",
		"distance": "30.05 astronomical units",
		"description": "Neptune is the farthest known planet from the Sun, an ice giant with strong winds and a deep blue color, similar to Uranus, due to methane in its atmosphere.",
		"title": "Neptune"
	}
]

Next, we’ll examine JSON results that contain hierarchical objects, in Recursive Schema.

Recursive Schema

As stated previously, schema can be designed to provide various levels of detail and complexity. For example, instead of generating a response listing ten items of detail, you may request that each item be optionally expanded to reveal relevant information about that particular item, and so on. In this section we’ll examine more complex schema.

About Tokens

Requests for more detailed responses requires more effort by the Language Model to generate tokens. Tokens are the basic units of text that language models process, which can be words, characters, or parts of words. They help the model understand and generate language by breaking down text into manageable pieces.

For some schema it may be prudent to adjust the number of tokens allowed by the operating system to be used by the AFM to enable it to better provide a complete response. The LanguageModel.GenerationOptions is used for this situation.

LanguageModel.GenerationOptions Class

This class is used to place limits on how the AFM repsonds.

new LanguageModel.GenerationOptions() → (LanguageModel.GenerationOptions) • The constructor for making a new instance.

maximumResponseTokens (Number or null) • (see below)

According to Apple’s documentation:

If the model produce maximumResponseTokens before it naturally completes its response, the response will be terminated early. No error will be thrown. This property can be used to protect against unexpectedly verbose responses and runaway generations. If no value is specified, then the model is allowed to produce the longest answer its context size supports. If the response exceeds that limit without terminating, an error will be thrown.

Our example schema example script will incorporate the use this class.

Multiple-Step Schema

For this example, we’ll ask the AFM to provide us an outline detailing the steps required to add solar power to a home. Each step in the outline can be expanded upon (as needed) to include it own “substeps” as well. To accomplish this optional expansion, the schema will reference itself (line 32) so it can be used recursively as a guide for the AFM to follow in constructing the expanded sections.

Multiple-Step Schema


{
    arrayOf: {
        name: "step-schema",
        properties: [
            {
                name: "title"
                isOptional: false
            },
            {
                name: "description",
                isOptional: false
            },
            {
                name: "priority",
                schema: {
                    name: "priority-schema",
                    anyOf: [
                        {
                            constant: "high"
                        },
                        {
                            constant: "low"
                        }
                    ]
                }
            },
            {
                name: "substeps",
                description: "A breakdown of steps.",
                isOptional: true,
                schema: {
                    arrayOf: {
                        referenceTo: "step-schema"
                    },
                    minimumElements: 1
                }
            }
        ]
    }
}

Here‘s a script using this schema:

Steps to Add Solar Power
  

(async () => {
	json = {arrayOf:{name:"step-schema",properties:[{name:"title",isOptional:false},{name:"description",isOptional:false},{name:"priority",schema:{name:"priority-schema",anyOf:[{constant:"high"},{constant:"low"}]}},{name:"substeps",description:"A breakdown of steps.",isOptional:true,schema:{arrayOf:{referenceTo:"step-schema"},minimumElements:1}}]}}
	schema = LanguageModel.Schema.fromJSON(json)
	prompt = "Provide a list of steps required to add solar power to a home."
	session = new LanguageModel.Session()
	options = new LanguageModel.GenerationOptions()
	options.maximumResponseTokens = 4096
	response = await session.respondWithSchema(prompt, schema)
	responseJSON = JSON.parse(response)
	console.log(JSON.stringify(responseJSON, null, 2))
})();

Here’s a typical result from the script:

Resulting JSON


[
    {
        "title": "Research and Planning",
        "substeps": [
            {
                "priority": "low",
                "title": "Assess Home Suitability",
                "description": "Evaluate your home's roof and location to determine potential for solar energy.",
                "substeps": [
                    {
                        "priority": "high",
                        "description": "Hire a professional to inspect your roof for solar panel installation potential.",
                        "title": "Roof Inspection"
                    },
                    {
                        "title": "Site Assessment",
                        "priority": "low",
                        "description": "Consider factors like sunlight exposure and shading."
                    }
                ]
            },
            {
                "priority": "low",
                "description": "Calculate energy needs and explore different system configurations.",
                "title": "Determine System Requirements"
            }
        ],
        "description": "Research available solar options and consult with a solar professional to determine the best system for your home.",
        "priority": "high"
    },
    {
        "title": "Get Permits and Approvals",
        "priority": "high",
        "substeps": [
            {
                "title": "Check Local Regulations",
                "description": "Understand zoning laws and building codes that affect solar installations.",
                "priority": "low",
                "substeps": [
                    {
                        "title": "Contact Local Authorities",
                        "description": "Reach out to local government offices for guidance.",
                        "priority": "high"
                    },
                    {
                        "description": "Complete and submit the required applications.",
                        "priority": "low",
                        "title": "Submit Permits"
                    }
                ]
            },
            {
                "title": "Obtain Necessary Approvals",
                "description": "Secure necessary approvals from relevant authorities.",
                "priority": "low"
            }
        ],
        "description": "Apply for necessary permits and work with local authorities if required."
    },
    {
        "description": "Choose the right solar panels and purchase them along with related equipment.",
        "title": "Select and Purchase Solar Panels",
        "substeps": [
            {
                "priority": "low",
                "title": "Research Panel Options",
                "description": "Compare different types, sizes, and efficiencies of solar panels.",
                "substeps": [
                    {
                        "description": "Learn about monocrystalline, polycrystalline, and thin-film technologies.",
                        "priority": "high",
                        "title": "Understand Technology"
                    },
                    {
                        "description": "Set a budget for purchasing panels and related equipment.",
                        "title": "Budgeting",
                        "priority": "low"
                    }
                ]
            },
            {
                "title": "Purchase Panels",
                "description": "Buy the selected panels and other necessary equipment.",
                "priority": "low"
            }
        ],
        "priority": "high"
    },
    {
        "substeps": [
            {
                "priority": "low",
                "substeps": [
                    {
                        "description": "Request multiple quotes from installation companies.",
                        "title": "Get Quotes",
                        "priority": "high"
                    },
                    {
                        "description": "Select the best-suited installation company based on quotes and reputation.",
                        "priority": "low",
                        "title": "Choose a Company"
                    }
                ],
                "title": "Hire Installation Company",
                "description": "Select and hire a reputable solar installation company."
            },
            {
                "title": "Schedule Installation",
                "priority": "low",
                "description": "Arrange a timeline for the installation process."
            }
        ],
        "title": "Install Solar Panels",
        "priority": "high",
        "description": "Have the solar panels installed on your home."
    },
    {
        "substeps": [
            {
                "substeps": [
                    {
                        "priority": "high",
                        "title": "Inspect Connections",
                        "description": "Check all electrical connections and systems."
                    },
                    {
                        "title": "Test Output",
                        "description": "Measure the system's output and performance metrics.",
                        "priority": "low"
                    }
                ],
                "description": "Conduct tests to ensure the system is functioning correctly.",
                "title": "System Testing",
                "priority": "low"
            },
            {
                "title": "System Optimization",
                "priority": "low",
                "description": "Adjust settings for maximum efficiency."
            }
        ],
        "title": "System Testing and Optimization",
        "description": "Test the installed system and optimize its performance.",
        "priority": "high"
    },
    {
        "description": "Link the solar system to the electrical grid for electricity distribution.",
        "title": "Connect to the Grid",
        "substeps": [
            {
                "priority": "low",
                "description": "Apply for and obtain necessary permits for grid connection.",
                "title": "Grid Connection Permit",
                "substeps": [
                    {
                        "priority": "high",
                        "title": "Submit Application",
                        "description": "Complete and submit the application for grid connection."
                    },
                    {
                        "priority": "low",
                        "description": "Wait for approval from utility companies.",
                        "title": "Wait for Approval"
                    }
                ]
            },
            {
                "description": "Set up all required hardware for grid connection.",
                "title": "Install Necessary Hardware",
                "priority": "low"
            }
        ],
        "priority": "high"
    },
    {
        "substeps": [
            {
                "title": "Regular Inspections",
                "priority": "low",
                "description": "Schedule regular inspections to check system health.",
                "substeps": [
                    {
                        "description": "Inspect electrical wiring for wear and damage.",
                        "priority": "high",
                        "title": "Check Wiring"
                    },
                    {
                        "description": "Track energy production and system efficiency.",
                        "priority": "low",
                        "title": "Monitor Performance"
                    }
                ]
            },
            {
                "description": "Perform routine maintenance to ensure system longevity.",
                "title": "Maintenance",
                "priority": "low"
            }
        ],
        "title": "Monitor and Maintain",
        "description": "Regularly monitor system performance and perform maintenance tasks.",
        "priority": "high"
    }
]

(⬇ see below ) The task hierarchy of the new project in OmniFocus matches that of the response JSON.

Solar Power Example Plug-In

The “AFM • Solar Power Example” plug-in demonstrates interacting with the AFM to create a new multi-level project in OmniFocus.

Plug-In Collection

For more interactive example plug-ins, demonstrating the use of the Apple Foundation Models, visit the Plug-In Collection page.