Apple Foundation Models

Welcome to a New Adventure in Automation

Apple’s Foundation Models Framework provides access to Apple’s on-device large language model that powers Apple Intelligence to help you perform intelligent tasks, triggered by natural languageA human written or spoken language as opposed to a computer language. text prompts.

Beginning with OmniFocus 4.8, on macOS 26, iOS 26, iPadOS 26, and visionOS 26, Omni Automation introduces support for querying Apple’s on-device Foundation Model directly from within its scripts and plug-ins. This is accomplished through the inclusion of the LanguageModel class.

LanguageModel Class

This class provides the mechanisms for querying Apple’s on-device Foundation Model using written natural language prompts and asynchronously returning its generated responses.

LanguageModel.Session

Communication between Omni Automation and the AFM (Apple Foundation Models) is done through “sessions.” The LanguageModel.Session class instantiates a new communication session with the AFM through the following constructor.

Constructor

• new LanguageModel.Session( ) → (LanguageModel.Session) • Creates a new session instance.

Instance Functions

Apple, and in turn Omni Automation, provides two functions for use with session instances, to make a query or request to the AFM: one for returning a human-readable text response, and one for returning structured data (schema).

• respond(prompt: String) → (Promise of String) • Produces a text response to a prompt.

• respondWithSchema(prompt: String, schema: LanguageModel.Schema, generationOptions: LanguageModel.GenerationOptions or null) → (Promise of String) • Produces a JSON response to a prompt using the provided schema.

We’ll examine the use of both functions, beginning with the respond(…) function that returns a text-based response, after which we’ll examine the respondWithSchema(…) function that returns structured data as its response.

Working with Text Responses

Here is an example of a simple natural-language prompt used with the respond(…) function to return a corresponding textual reply:

Querying the Apple Foundation Models
OmniFocusOmniGraffleOmniOutlinerOmniPlan  

(async () => {
	try {
		prompt = "What is “Apple Intelligence?”"
		session = new LanguageModel.Session()
		response = await session.respond(prompt)
		console.log(response)
	}
	catch(err){
		new Alert(err.name, err.message).show()
	}
})();

And the response logged to the console:

Apple Foundation Models Response
Copy

"Apple Intelligence" refers to a suite of artificial intelligence features that Apple plans to integrate into its products and services. While specific details and features might evolve, generally, Apple Intelligence is expected to enhance user experiences across Apple devices by leveraging AI to improve tasks such as productivity, personalization, and interaction with apps.

Potential features might include smarter Siri interactions, enhanced data analysis in apps like Mail or Photos, predictive capabilities in search functions, and improved learning algorithms for personalization. Apple tends to emphasize privacy and security in its AI initiatives, so these features are likely designed to work seamlessly within the Apple ecosystem while respecting user data.

For the most accurate and up-to-date information, it's best to check Apple's official announcements or their press releases related to new product updates.

IMPORTANT: The query and response are performed asynchronously A script will pause while it waits for the response. and so may take some time to complete.

Prompt Tester Plug-In

It may take practice with the Apple Foundation Models (AFM) to develop the type of prompts that will produce the data you desire. Use this Prompt Tester Plug-In to assist you in the development of your prompts. With this plug-in, the AFM responses to your entered prompts are logged to the Omni application Automation Console window.

Prompt Tester Plug-In (Requires: macOS 26+, iOS 26+, iPadOS26+, or visionOS 26+)
OmniFocusOmniGraffleOmniOutlinerOmniPlan 

(⬇ see below ) Prompt Tester plug-in on the Automation Console toolbar:

As another example, let’s ask the AFM for a list of the names of the planets in our solar system. Our query or “prompt”A Prompt is a text input that the model responds to. will reflect our request for information:

“List the names of the planets in our solar system.”

AFM Interaction
OmniFocusOmniGraffleOmniOutlinerOmniPlan  

(async () => {
	try {
		prompt = "List the names of the planets in our solar system."
		session = new LanguageModel.Session()
		response = await session.respond(prompt)
		console.log(response)
	}
	catch(err){
		new Alert(err.name, err.message).show()
	}
})();

NOTE: With Language Models, the response to a specific prompt may vary with each request. Here’s a typical response to the previous query:

Typical Response
Copy

The planets in our solar system, in order from closest to the Sun to farthest, are:

1. Mercury
2. Venus
3. Earth
4. Mars
5. Jupiter
6. Saturn
7. Uranus
8. Neptune

Note that the response is returned as a block of text (String) that is written to be conversant or conversational, as if the response was generated by a human. It is assumed by the AFM that the response will be read by a person or agent.

If a response in textual format is adequate for your purposes, it is fairly easy to use Omni Automation to add the response to an outgoing email or copy it to the clipboard.

Copy and Send AFM Response
OmniFocusOmniGraffleOmniOutlinerOmniPlan  

(async () => {
	try {
		prompt = "What is “Apple Intelligence?”"
		session = new LanguageModel.Session()
		response = await session.respond(prompt)
		console.log(response)
		
		Pasteboard.general.string = response
		
		email = new Email()
		email.subject = "AFM Query"
		email.body = response
		email.generate()
	}
	catch(err){
		new Alert(err.name, err.message).show()
	}
})();

However, if your automation requires extracting specific data from the text response, read on.

If you look at the previous example text response…

The planets in our solar system, in order from closest to the Sun to farthest, are:

1. Mercury
2. Venus
3. Earth
4. Mars
5. Jupiter
6. Saturn
7. Uranus
8. Neptune

…it’s easy for you to quickly pick out the names of the planets, as our brains are designed to recognize patterns and order. However, scripts don’t “think,” so if you intend for the data in the response to be absorbed by a script, you’ll need to include code in your script for locating and retrieving the desired data that has been mixed-in with the rest of the response text. This can be challenging since no two responses from the AFM may be organized the same!

Fortunately, there’s a solution for this issue: Structured Data

Structured Data

Structured Data is information formatted in a way that can be predictably parsed by scripts and applications. The Apple Foundation Models (AFM) supports the inclusion of multiple types of structured data in model text responses, some of which are:

• JSON • (JavaScript Object Notation) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of name–value pairs and arrays. Since Omni Automation is based upon JavaScript Core, it includes the standard functions for parsing JSON data.
• XML • (Extensible Markup Language) is a markup language and file format by the W3C for storing, transmitting, and reconstructing data. By default, Omni Automation includes XML Classes for writing and parsing XML.

Your prompts can include a request for the data to be returned as structured content placed within the text response.

So if we want Structured Data returned in the responses from the Apple Foundation Models, we have to explicitly request it in the prompt passed to the LanguageModel.Session instance.

Requesting structured data in JSON format:

Requesting Structured Data (JSON)
OmniFocusOmniGraffleOmniOutlinerOmniPlan  

(async () => {
	try {
		prompt = "Provide, as a JSON array, the names of the planets in our solar system."
		session = new LanguageModel.Session()
		response = await session.respond(prompt)
		console.log(response)
	}
	catch(err){
		new Alert(err.name, err.message).show()
	}
})();

typical response:

Response Containing Structured Data (JSON)
Copy

Certainly! Here is a JSON array containing the names of the planets in our solar system:

```json
[
    "Mercury",
    "Venus",
    "Earth",
    "Mars",
    "Jupiter",
    "Saturn",
    "Uranus",
    "Neptune"
]
```

Requesting structured data in XML format:

Requesting Structured Data (XML)
OmniFocusOmniGraffleOmniOutlinerOmniPlan  

(async () => {
	try {
		prompt = "Provide in XML format using the element names of “planets” and “planet” the names of the planets in our solar system."
		session = new LanguageModel.Session()
		response = await session.respond(prompt)
		console.log(response)
	}
	catch(err){
		new Alert(err.name, err.message).show()
	}
})();

typical response:

Response Containing Structured Data (XML)
Copy

Certainly! Below is an XML representation of the planets in our solar system using the specified element names:

```xml
<planets>
    <planet>Mercury</planet>
    <planet>Venus</planet>
    <planet>Earth</planet>
    <planet>Mars</planet>
    <planet>Jupiter</planet>
    <planet>Saturn</planet>
    <planet>Uranus</planet>
    <planet>Neptune</planet>
</planets>
```

This XML structure lists each planet under the `<planet>` element, followed by the planet's name enclosed in angle brackets.

Since the structured data within the response will be placed within back-ticks, it will only require a few lines of standard JavaScript to extract it:

Extracting Structured Data from Text
Copy

typeID = "json" // "xml"
openingTag = "```" + typeID
start = response.indexOf(openingTag) + openingTag.length
end = response.indexOf("```", start)
dataText = response.substring(start, end).trim()

returning:

Structured Data from Response (JSON)
Copy

JSON.parse(dataText)
//--> ["Mercury","Venus","Earth","Mars","Jupiter","Saturn","Uranus","Neptune"]

Here’s a complete script demonstrating how to query and parse the textual response to retrieve data:

Parsing the Apple Foundation Models Response (JSON)
Copy

(async () => {
	try {
		// QUERY THE AFM (Apple Foundation Models)
		prompt = "Provide, as a JSON array, the names of the planets in our solar system."
		session = new LanguageModel.Session()
		response = await session.respond(prompt)

		// EXTRACT THE STRUCTURED DATA FROM TEXT RESPONSE
		openingTag = "```json"
		start = response.indexOf(openingTag) + openingTag.length
		end = response.indexOf("```", start)
		dataText = response.substring(start, end).trim()

		// CONVERT EXTRACTED TEXT INTO JSON OBJECT
		json = JSON.parse(dataText)
		
		// CODE FOR CREATING CONTENT IN OMNI APP USING STRUCTURED DATA 
	}
	catch(err){
		new Alert(err.name, err.message).show()
	}
})();

Working with Parsed XML

Here’s how to work with XML parsed from a response:

Structured Data from Response (XML)
Copy

"<planets>
    <planet>Mercury</planet>
    <planet>Venus</planet>
    <planet>Earth</planet>
    <planet>Mars</planet>
    <planet>Jupiter</planet>
    <planet>Saturn</planet>
    <planet>Uranus</planet>
    <planet>Neptune</planet>
</planets>"

And a few lines of code using the built-in Omni Automation XML Class will convert the structured data string into a parsable XML data object:

Converting Strings into XML Data
Copy

behavior = new XML.WhitespaceBehavior(XML.WhitespaceBehavior.Type.Ignore)
data = Data.fromString(dataText)
xmlDoc = XML.Document.fromData(data, behavior)
console.log(xmlDoc.rootElement.name)
//--> "planets"
planetNames = xmlDoc.rootElement.children.map(item => item.stringContents)
//--> ["Mercury", "Venus", "Earth", "Mars", "Jupiter", "Saturn", "Uranus", "Neptune"]

Once the structured data has been extracted from the response text, it can be used to create content in Omni applications, such as a project with tasks in OmniFocus!

Here’s the complete script for querying the AFM and extracting the structured XML data from the response text:

Parsing the Apple Foundation Models Response (XML)
OmniFocusOmniGraffleOmniOutlinerOmniPlan  

(async () => {
	try {
		// QUERY THE AFM (Apple Foundation Models)
		prompt = "Provide in XML format using the element names of “planets” and “planet” the names of the planets in our solar system."
		session = new LanguageModel.Session()
		response = await session.respond(prompt)

		// EXTRACT THE STRUCTURED DATA FROM TEXT RESPONSE
		openingTag = "```xml"
		start = response.indexOf(openingTag) + openingTag.length
		end = response.indexOf("```", start)
		xmlText = response.substring(start, end).trim()

		// CONVERT EXTRACTED TEXT INTO AN XML DOCUMENT
		behavior = new XML.WhitespaceBehavior(XML.WhitespaceBehavior.Type.Ignore)
		data = Data.fromString(xmlText)
		xmlDoc = XML.Document.fromData(data, behavior)

		// PARSE THE XML
		planetNames = xmlDoc.rootElement.children.map(item => item.stringContents)
		console.log(planetNames)
		//--> ["Mercury", "Venus", "Earth", "Mars", "Jupiter", "Saturn", "Uranus", "Neptune"]
		
		// CODE FOR CREATING CONTENT IN OMNI APP USING STRUCTURED DATA 
	}
	catch(err){
		new Alert(err.name, err.message).show()
	}
})();

Summary

Although the textual response from the respond(…) function can be parsed to retrieve specific data, there is a much better mechanism available to return just the data your automation requires. This is done through the use of user-provided Schema.