Apple Foundation Models: AI Tools

For developers of Omni Automation plug-ins, the new “AI Tools” feature enables the Apple Foundation Models framework to execute custom code snippets or “AI Tools” that you create and add to your plug-in or library. Think of “AI Tools” as JavaScript functions within your code that can be called as needed by the Apple Foundation Models frameworks.

LanguageModel.Tool Class (new)

When working with a LanguageModel, you can provide tools that it can use to obtain information or make calculations that haven’t been provided through its training data or the current session’s prompt history.

Constructor

Here’s the code for creating a new instance of the LanguageModel.Tool class:

New Language Model Tool
Copy

new LanguageModel.Tool(
	name: String, 
	description: String, 
	inputSchema: LanguageModel.Schema or null, 
	function: Function
) → (LanguageModel.Tool) 

• name — Each tool must be given a unique name, that is used by the language model both to understand what the tool does and to invoke the tool when it wants to use it.
• description — The tool’s description helps the language model understand when and how to use the tool.
• schema — The provided schema specifies how the tool would like its input to be formatted. If no input schema is provided, the tool will use a simple default schema with a text property.
• function — The provided function takes input (based on the provided description and schema) and calculates a result to return back to the language model. The result will be encoded as JSON and returned to the model for use in its calculations. Note that the function can be asynchronous but doesn’t have to be.

LanguageModel.Session Class (update)

The Language.Model class has been updated to include a new class function for instantiating a session that uses one or more tools:

Class Functions

• withTools(tools: Array of LanguageModel.Tool, instructions: String or null) → (LanguageModel.Session) • Creates a new session instance, providing tools.

Creating an AI Tool

For example, here’s a function that instantiates an “AI Tool” that the Apple Foundation Models framework (AFM) can call to get the text content currently on the general clipboard (also called the ”PasteBoard”):

“Get Clipboard Text AI Tool”
Copy

const getClipboardTextTool = new LanguageModel.Tool(
	"getClipboardText",
	"This tool is used to extract the text from the general clipboard.",
	null,
	async () => {
		console.log("getClipboardText…");
		const clipboardText = Pasteboard.general.string;
		console.log(clipboardText);
		return clipboardText;
	}
);

Here’s a function, that can be added to an Omni Automation library, that returns a reference to the instantiated tool:

Function for Generating “Get Clipboard Text AI Tool”
Copy

const getClipboardTextTool = function() {
	console.log("getClipboardTextTool() function called");
	
	const toolObject = new LanguageModel.Tool(
		"getClipboardText",
		"This tool is used to extract the text from the general clipboard.",
		null,
		async () => {
			console.log("getClipboardText…");
			const clipboardText = Pasteboard.general.string;
			console.log(clipboardText);
			return clipboardText;
		}
	);
	
	console.log("“getClipboardText” tool created");
	return toolObject;
};

When this “AI Tool” is added to a plug-in, it can be invoked by the AFM during the execution of the plug-in code, and the resulting textual content used by the AFM in performing its assigned task.

In the following example plug-in the AFM is instructed to use the tool to retrieve and log the text on the clipboard.

[ IMPORTANT: This plug-in currently works with only OmniFocus 4.8 or OmniOutliner 6.0. ]

Using an AI Tool to Log the Clipboard Text
OmniFocusOmniGraffleOmniOutlinerOmniPlan  

(async () => {
	try {
		const getClipboardTextTool = new LanguageModel.Tool(
			"getClipboardText",
			"This tool is used to extract the text from the general clipboard.",
			null,
			async () => {
				console.log("getClipboardText…");
				const clipboardText = Pasteboard.general.string;
				console.log(clipboardText);
				return clipboardText;
			}
		);
		
		const instructions = "Use the getClipboardText tool to retrieve the textual contents of the clipboard.";

		const session = LanguageModel.Session.withTools(
			[getClipboardTextTool],
			instructions
		);
		
		const resultSchema = LanguageModel.Schema.fromJSON({
			name: "result",
			properties: [
				{
					name: "text",
					description: "The text resulting from the prompt"
				}
			]
		});

		const prompt = "What text is on the clipboard?";

		const options = new LanguageModel.GenerationOptions();
		options.maximumResponseTokens = 4096;

		const responseStr = await session.respondWithSchema(
			prompt,
			resultSchema,
			options
		);
		
		console.log(responseStr);
		const responseObj = JSON.parse(responseStr);
		console.log(responseObj.text);
	}
	catch(err){
		new Alert(err.name, err.message).show();
	}
})();

OmniOutliner Plug-In

The “Outline the Clipboard” plug-in for OmniOutliner uses an AI Tool to scan and summarize the clipboard text to generate a series of summary points that become rows in an OmniOutliner outline.

Omni AI Tools Library (@)

To make it easier to access the functionality of the Apple Foundation Models framework, we’ve created a library of AI Tools that can be used by all Omni applications (currently supporting AFM access).

Omni Automation Libraries are specialized plug-ins containing functions that can be called by the active plug-ins of the host Omni application. Such libraries are stored in the same locations as “regular” plug-ins, and loaded by plug-ins using code similar to the following example:

Loading the Omni AI Tools Library
Copy

// LOCATE AND LOAD THE OMNIFOCUS AI TOOLS LIBRARY
libraryID = "com.omni-automation.all.ai-tools-library"
libraryTitle = "Omni AI Tools Library"
libraryPlugIn = PlugIn.find(libraryID)
if (!libraryPlugIn){throw new Error(`The “${libraryTitle}” plug-in is not installed.`)}
library = libraryPlugIn.library(libraryPlugIn.displayName)

Once a library has been “loaded,” its functions may be called within the code of the loading plug-in. Line 11 of the following example, instantiates an AI Tool for retrieving the text of the clipboard, by calling a function from the loaded library:

Using Omni AI Library to Retrieve Clipboard Text
OmniFocusOmniGraffleOmniOutlinerOmniPlan   


(async () => {
	try {
		// LOCATE AND LOAD THE OMNIFOCUS AI TOOLS LIBRARY
		const libraryID = "com.omni-automation.all.ai-tools-library";
		const libraryTitle = "Omni AI Tools Library";
		const libraryPlugIn = PlugIn.find(libraryID);
		if (!libraryPlugIn) { throw new Error(`The “${libraryTitle}” plug-in is not installed.`); }
		const library = libraryPlugIn.library(libraryPlugIn.displayName);
		
		// INSTANTIATE AN AI TOOL FROM LIBRARY
		const getClipboardTextTool = library.getClipboardTextTool();

		// CREATE A NEW SESSION USING THE AI TOOL
		const session = LanguageModel.Session.withTools(
			[getClipboardTextTool],
			null
		);

		// THE OUTPUT SCHEMA FOR RETURNING THE EXTRACTED TEXT
		const resultSchema = LanguageModel.Schema.fromJSON({
			name: "result",
			properties: [
				{
					name: "text",
					description: "The text resulting from the prompt"
				}
			]
		});

		// THE PROMPT FOR THE AFM
		const prompt = "Use the getClipboardText tool to retrieve the textual contents of the clipboard.";

		// MAXIMIZING THE TOKENS USED TO COMPUTE
		const options = new LanguageModel.GenerationOptions();
		options.maximumResponseTokens = 4096;

		// ACTIVATING THE SESSION
		const responseStr = await session.respondWithSchema(
			prompt,
			resultSchema,
			options
		);

		// RETRIEVING THE TEXT FROM THE JSON RESPONSE OBJECT
		console.log(responseStr);
		const responseObj = JSON.parse(responseStr);
		console.log(responseObj.text);
	}
	catch (err) {
		new Alert(err.name, err.message).show();
	}
})();