January 18: Support for content publishing, LLM tools, CLIP image embeddings, bug fixes

New Features

• 💡 Graphlit now supports content publishing, where documents, audio transcripts and even image descriptions, can be summarized, and repurposed into blog posts, emails or AI-generated podcasts. With the new publishContents mutation, you can configure LLM prompts for summarization and publishing, and assign specifications to use different models and/or system prompts for each step in the process. The published content will be reingested into Graphlit, and can be searched or used for conversations, like any other form of content.

• 💡 Graphlit now supports publishing conversations as content with the new publishConversation mutation. You can generate text or audio transcripts of your conversations, to be reused in other tools.

• 💡 Graphlit now supports bulk summarization of contents with the summarizeContents mutation. You can filter a set of content, by feed, by observable or by similar text, and run a set of summarizations across each content in parallel.

• 💡 Graphlit now supports LLM entity extraction, with the new MODEL_TEXT entity extraction service type. Similar to using Azure Cognitive Service Text Analytics, you can use any OpenAI or Anthropic model for extracting entities from text. Internally the LLM returns JSON-LD entities, which we convert into Person, Organization, Place, etc. entities and assign observations to the extracted content.

• 💡 Graphlit now supports LLM tools (aka function calls) with OpenAI models. You can define the tools to be used with the LLM in the specification object. With the new extractContents mutation, you can execute a prompt against content using a specification with tools defined. The mutation will return the JSON arguments assigned by the LLM.

• 💡 Graphlit now supports callback webhooks for LLM tools. If you assign a URI in the ToolDefinition object, Graphlit will call your webhook the tool name and JSON arguments. When you respond to the webhook with JSON, we will add that response to the LLM messages, and ask the LLM to complete the original prompt.

• 💡 Graphlit now supports the selection of the Deepgram model (such as Meeting, Phonecall or Finance) with the preparation workflow. Also, you can assign your own Deepgram API key, which will be used for audio transcription using that workflow.

• Added support for CLIP image embeddings using Roboflow, which can be used for similar image search. If you search for contents by similar contents, we will now use the content's text and/or image embeddings to find similar content.

• Added support for dynamic web page ingestion. Graphlit now navigates to and automatically scrolls web pages using Browserless.io, so we capture the fully rendered HTML before extracting text. Also, we now support web page screenshots, if enabled with enableImageAnalysis property in preparation workflow. These screenshots can be analyzed with multimodal modals, such as GPT-4 Vision, or can be used to create image embeddings for similar image search.

• Added table parsing when preparing documents. We now store structured (tab-delimited) text in the JSON text mezzanine which is extracted from documents in the preparation workflow.

• Added reverse geocoding of lat/long locations found in image or other content metadata. We now store the real-world address with the content metadata, for use in conversations.

• Added assistant messages to the conversation message history provided to the LLM. Originally we had included only user messages, but now we are formatting both user and assistant messages into the LLM prompt for conversations.

• Added new chunking algorithm for text embeddings. We support semantic chunking at the page or transcript segment level, and now will create embeddings from smaller sized text chunks per page or segment.

• Added content metadata to text and image embeddings. To provide better context for the text embeddings, we now include formatted content metadata, which includes fields like title, subject, author, or description. For emails, we include to, from, cc, and bcc fields.

• Added helper mutations isContentDone and isFeedDone which can be used for polling completion of ingested content, or all content ingested by a feed.

• Added richer image descriptions generated by the GPT-4 Vision model. Now these provide more useful detail.

• Added validation of extracted hyperlinks. Now we test the URIs and remove any inaccessible links during content enrichment.

• Added deleteContents, deleteFeeds, and deleteConversations mutations for multi-deletion of contents, feeds or conversations.

• Added deleteAllContents, deleteAllFeeds, and deleteAllConversations mutations for bulk, filtered deletion of entities. You can delete all your contents, feeds, or conversations in your project, or a filtered subset of those entities.

• ℹ️ Starter tier now has a higher content limit of 100K content items.

• ⚡ In the OpenAIImageExtractionProperties type, the detailMode field was renamed to detailLevel.

• ⚡ Each SummarizationStrategy object now accepts the specification which is used by the summarization, rather than being assigned at the preparation workflow stage.

• ⚡ addCollectionContents and removeCollectionContents mutations have been deprecated in favor of addContentsToCollections and removeContentsFromCollection mutations.

Bugs Fixed

• GPLA-1846: Parse Markdown headings into mezzanine JSON

• GPLA-1779: Not returning SAS token with mezzanine, master URIs

• GPLA-1348: Summarize text content, not just file content

• GPLA-1297: Not assigning content error message on preparation workflow failure