Omni Automation Video

(0:00) (TOP) AI for the Rest of Us Meets the Tools that Power Us

(0:07) (TOP) Introducing Apple Intelligence and Omni Automation

(0:14) (TOP) Beginning with macOS 26, iOS 26, iPadOS 26, and VisionOS 26, software developers now have direct access to the on-device Foundation Models at the core of Apple Intelligence. This is the same language model that powers machine learning and Apple's writing tools.

(0:36) (TOP) In turn, Omni will enable Omni Automation, using natural language prompts, to directly communicate with the on-device Foundation Models and to receive responses in either textual or JSON structured format.

(0:53) (TOP) Beginning with the release of OmniFocus 4.8, Omni Automation plugins can incorporate the use of the Apple Foundation Models to assist in the creation of content. Let's examine how to use Omni Automation to interact with Apple's On Device Foundation Models.

(1:09) (TOP) Responses from queries to the AFM can be as either text or structured JSON data. Let's begin with how to elicit text responses.

(1:20) (TOP) In this example, the JavaScript code for communicating with the Apple Foundation Models, or AFM, is placed within a self-executing, anonymous, asynchronous function.

(1:30) (TOP) Responses from the Foundation Models require varying lengths of time to be generated, and so the supporting code must be asynchronous in its design.

(1:39) (TOP) Communication with the AFM begins with the creation of a prompt. A prompt is a question, or set of instructions to the Foundation Models, that provide it the context necessary to generate a response, relevant to what you are asking or requesting.

(1:54) (TOP) In this example, the prompt is the simple question, what is Apple Intelligence?

(2:00) (TOP) Next, a new session with the AFM is instantiated using the Language Model Session constructor.

(2:09) (TOP) The prompt is then passed to the AFM as a parameter of the respond instance function of the Language Model class. Note the use of the await operator, telling the script to wait for the response from the Foundation Models.

(2:24) (TOP) And lastly, the returned text response is logged to the automation console window.

(2:32) (TOP) By default, the resultant text is conversant in nature, possibly verbose, and may vary with each instance.

(2:39) (TOP) For the efficient retrieval of data, scripts and plugins can request responses as structured JSON by providing the Foundation Models an example schema of the desired data format.

(2:52) (TOP) Let's examine how to use a JSON schema to return an array of strings.

(2:58) (TOP) In terms of the Apple Foundation Models, what is a schema? A schema is an example data structure written in JavaScript object notation or JSON demonstrating to the AFM the manner in which you wish its generated data be returned from your query.

(3:16) (TOP) This expanded example, is a schema, titled “StringArraySchema” that has an array of properties as its value. The properties array contains a single property, named result, whose value will be returned as a simple array of text strings.

(3:31) (TOP) Here's the same schema, condensed without its superfluous spacing.

(3:39) (TOP) And here's the same schema, used in an interaction with the Apple Foundation Models.

(3:45) (TOP) The asynchronous function begins by converting the provided schema into a schema object used by the AFM by passing the JSON as the parameter for the “fromJSON” instance function of the “LanguageModel” schema class.

(4:01) (TOP) The prompt in this example is asking for a list of the names of the planets in the solar system, while the provided schema is indicating that the data be returned as an array of strings.

(4:13) (TOP) A new session with the Foundation Models is instantiated and then both the prompt and its corresponding schema object are passed to the AFM as parameters in the “respondWithSchema” instance function of the “LanguageModel” class.

(4:28) (TOP) And finally, the resulting response is converted from a string into a JSON object and the value of the result property is logged to the automation console window.

(4:40) (TOP) The data returned by the Foundation Models and logged to the console will be a simple array of the planet names.

(4:48) (TOP) This concludes a short introductory overview of using Omni Automation to interact with the on-device Apple Foundation Models.

(5:00) (TOP) For detailed documentation with example plugins, please visit omni-automation.com/shared/alm.html Thank you.