# April 26: Support for fact evidence citations, targeted distribution, agent modes

### New Features

* 💡 Graphlit now supports **fact evidence citations** via `FactEvidence` and `FactCitation`, making it easier to trace extracted facts back to supporting content or conversations.
* 💡 We have added **targeted distribution** support with `DistributionTarget`, operation/kind enums, and provider-specific properties for Notion, Google Drive, OneDrive, Confluence, Slack, Gmail, Outlook, HubSpot, Salesforce, Attio, Google Calendar, Microsoft Calendar, Linear, Jira, Zendesk, Intercom, Google Docs, Microsoft Word, SharePoint, Discord, Teams, X, GitHub, GitLab, and LinkedIn.
* We have expanded **Agent APIs** with durable modes, commands, heartbeat settings, research depth, persona/specification references, content filters, and distribution targets.
* We have added **GPT Image-2** and new image publishing controls for quality, width, height, output format, compression, and moderation.
* We have added **LinkedIn distribution and search** configuration for publishing LinkedIn posts and filtering LinkedIn search by date range.
* We have added additional distribution discovery types, including `IssueDistributionDiscoveryResults`, to support more precise issue-targeted publishing workflows.

### Bugs Fixed

* Improved Gmail feed timeout recovery for newly created feeds with large mailboxes.
* Fixed image publishing prompt compilation so generated image requests preserve the intended prompt structure.
* Improved Notion parsing resilience by preserving unknown block, property, file, parent, and rich text object types instead of failing deserialization.


# April 19: Support for Evernote, replicas, skill feeds

### New Features

* 💡 Graphlit now supports **Evernote feeds**, including OAuth support and Evernote note/notebook ingestion.
* 💡 We have added **Replica APIs** for syncing content, conversations, and skills into external artifact stores, including Git-backed Markdown and WebVTT outputs.
* We have added **Skill feeds** for installing skill packages from GitHub or GitLab, with preview support for new, updated, unchanged, and invalid skills.
* We have added **on-demand content classification** result types for classifying content outside the normal ingestion workflow.
* We have added **content Markdown export** support through `markdown(maxTokens)`, `markdownUri`, and content snippets.
* We have improved **CSV table embedding** support for better retrieval over structured tabular content.
* We have added **conversation criteria** inputs for more targeted conversation queries and agent workflows.

### Bugs Fixed

* Improved Notion client serialization coverage for newer and unknown Notion block types.
* Improved summarization instruction caching for repeated summarization workflows.
* Improved external content read behavior for connector-backed feeds.


# April 10: Support for X post ingestion, Reddit content, OAuth connectors

### New Features

* 💡 Graphlit now supports **X post ingestion** through URL-native X content helpers and metadata/search support.
* 💡 We have added **Reddit content helpers** for richer post ingestion and preparation workflows.
* We have added connector-scoped **OAuth helpers** for Slack, X, GitHub, GitLab, Atlassian, and Notion.
* We have added `read` result APIs for externally reading content through connector-backed sources.
* We have expanded feed and connector sanitization work so connector configuration can be safely reused across workflow surfaces.

### Bugs Fixed

* Improved OAuth connector scoping to avoid cross-connector credential ambiguity.
* Improved package and site feed tests around externally read content.


# March 25: Support for GitLab feeds, waterfall enrichment, agent bureaus

### New Features

* 💡 Graphlit now supports **GitLab feeds** for files, issues, merge requests, commits, and milestones through new GitLab connector and feed service types.
* 💡 We have added **waterfall entity enrichment**, allowing enrichment to cascade across providers until the requested completeness level is reached.
* We have added **Bureau**, **Desk**, and **Skill** as top-level entity APIs for organizing Agent workspaces, instructions, and reusable capabilities.
* We have added **channel connectors** for Slack, Microsoft Teams, Discord, Telegram, WhatsApp, and Google Chat, giving agents and workflows a typed channel surface.
* We have added organization and user **credit quota** APIs for project-level billing controls.
* We have added **Nyne** entity enrichment support.

### Bugs Fixed

* Improved GitHub and GitLab OAuth connector handling across file, commit, issue, and pull request feeds.
* Improved enrichment completeness checks for multi-provider enrichment workflows.


# March 8: Support for feed preview, Productlane, initiative feeds

### New Features

* 💡 Graphlit now supports **feed preview** APIs, allowing developers to inspect candidate feed items before ingestion.
* 💡 We have added **Productlane** integration across CRM and issue-style workflows, including Productlane threads, issues, objects, changelogs, and articles.
* We have added **initiative feeds** with new `Initiative` connector and result types for GitHub, GitLab, Jira, and Linear planning objects.
* We have added date range filtering to more feed property types with `beforeDate` and `afterDate` support.
* We have expanded calendar and file listing APIs with new result types for Google Drive drives, Atlassian sites, Jira projects, and calendar events.

### Bugs Fixed

* Improved feed preview coverage across connector-backed feeds.
* Improved calendar event handling for Google and Microsoft calendar services.


# February 27: Support for Crustdata, agents, Google Veo video

### New Features

* 💡 Graphlit now supports **Crustdata** for company and person discovery, enrichment, and watcher-style signals through new feed, enrichment, and search service APIs.
* 💡 We have added the first public **Agent** entity APIs, including agent input, output, filtering, and result types for agent-driven workflows.
* We have added **Google Veo video generation** support for video publishing workflows.
* We have added **Google Nano Banana 2** image model support.
* We have added draft **distribution** support through new distribution connector and distribution result types.
* We have added conversation collections for organizing generated and ingested conversation content.

### Bugs Fixed

* GPLA-4849: Email ingestion now parses complete message bodies more reliably.
* GPLA-4852: Hybrid search can now find videos by name.
* GPLA-4851: Improved `retrieveSources` latency for larger retrieval requests.
* Fixed SAS URI handling and output filenames for generated video assets.


# February 8: Support for Notion OAuth, Opus 4.6, conversation extraction

### New Features

* 💡 Graphlit now supports **Notion OAuth**, allowing Notion feeds to connect through OAuth-backed connectors instead of only static credentials.
* 💡 We have added support for **Anthropic Opus 4.6** with a 1M context window.
* Conversation content now supports **entity and fact extraction**, so ingested support conversations, meeting transcripts, and chat-style content can populate the knowledge graph.
* Intercom, Zendesk, and Notion feeds now generate `PAGE` content where appropriate, improving downstream preparation and retrieval behavior for long-form source material.
* We have added **Persona** as a top-level entity for modeling user, customer, or audience archetypes used by Agent workflows.

### Bugs Fixed

* GPLA-4840: Reddit subreddit names are now sanitized before feed creation.
* GPLA-4843: Microsoft Teams messages now include embedded images during ingestion.
* GPLA-4841: Entity relationship creation now preserves valid source entity identifiers.
* GPLA-4842: Person updates now retain the correct project identifier.


# January 29: Support for Confluence, HubSpot conversations, Asana and Monday.com

### New Features

* 💡 Graphlit now supports **Confluence feeds**. Developers can enumerate Atlassian sites and Confluence spaces, then ingest pages from selected workspaces.
* 💡 We have added **HubSpot Conversations** and **HubSpot Tickets** as feed sources, extending HubSpot ingestion beyond CRM objects into support and messaging workflows.
* We have added **Asana** and **Monday.com** issue feeds, allowing project and task data to be ingested alongside GitHub, Jira, Linear, and other issue sources.
* We have added **Krisp meeting transcripts** as a meeting data source.
* We have added **ElevenLabs Scribe V2** transcription support for file preparation workflows.
* We have expanded conversation storage so conversation messages can be stored, queried, and used for entity and fact extraction.

### Bugs Fixed

* GPLA-4821: Slack attachment filenames are now sanitized before ingestion.
* GPLA-4830: Fact queries now return extracted facts correctly in Dossium.
* GPLA-4829: Improved full web page ingestion latency.
* GPLA-4789: Improved handling for Stripe rate limits during billing operations.


# January 12: Support for S3-compatible storage, Slack webhooks

### New Features

* 💡 Graphlit now supports **S3-compatible file services** including Cloudflare R2, Digital Ocean Spaces, MinIO, and Wasabi. Configure the `customEndpoint` property in `AmazonFeedProperties` for both file and site feed ingestion.
* We have added support for **Slack webhook feed ingestion** using the `MONITOR` sync policy type. Feeds can now accept Slack messages via webhook for real-time ingestion.

### Bugs Fixed

* GPLA-4808: Fixed markdown extraction failures for documents with deeply nested elements or large tables
* GPLA-4811: Fixed "contentParts cannot be empty collection" errors when processing certain message sequences
* GPLA-4776: Fixed event storage failures caused by invalid project identifier validation


# January 3: Support for Fireflies.ai, Attio meetings, Fathom transcripts

### New Features

* 💡 We have added **Fireflies.ai** as a new meeting transcript data source via `FirefliesFeedProperties`, enabling ingestion of meeting transcripts and recordings directly into your knowledge graph.
* We have added **Attio meeting transcripts** as a data source via `AttioMeetingProperties`. You can filter meetings by `afterDate` and `beforeDate` to control ingestion scope.
* We have added **Fathom** as a meeting transcript data source via `FathomProperties` and the `FATHOM` feed service type.

### Bugs Fixed

* GPLA-4806: Slack feeds now correctly ingest messages from threads and thread replies posted to channels


# December 31: Support for tool calling in prompt() mutation, user-based memory, fact extraction

### New Features

* 💡 Graphlit now supports **tool calling in the `prompt` mutation**. You can pass a `tools` argument to enable function calling within single-turn prompts, with tool call responses properly returned in the completion.
* 💡 We have introduced **Facts** as a first-class entity type. Facts are extracted assertions from content (including memories) with entity mentions, text spans, and temporal validity (`validAt`, `invalidAt`). Facts support evidence linking back to source content and can be queried independently via the `facts` query.
* We have added an **Agentic specification type** to differentiate specifications used for `promptConversation` vs `streamAgent`, enabling better conversation processing within agentic contexts.
* We have added support for **Slack OAuth connector**, allowing user-based authentication in addition to bot tokens.

### Bugs Fixed

* GPLA-4795: Fixed content enrichment errors caused by invalid URI parameters
* GPLA-4797: Fixed fact extraction failures from text documents with null values
* GPLA-4799: Fixed pagination errors when skip parameter exceeds 100,000
* GPLA-4794: Fixed null reference errors when parsing certain Slack messages


# December 17: Support for Gemini 3 Flash, Cohere Rerank 4

### New Features

* 💡 Graphlit now supports **Google Gemini 3 Flash Preview** with the model enum `GEMINI_3_FLASH_PREVIEW`.
* 💡 We have added support for **Cohere Rerank 4** for improved retrieval relevance scoring.
* We have added support for **OpenAI GPT-Image 1.5** for image generation.

### Bugs Fixed

* GPLA-4785: Box refresh tokens are now properly saved after access token generation, preventing feed authentication failures
* GPLA-4783: Fixed null reference errors when processing inline file processor connectors
* GPLA-4780: Fixed function timeout exceptions on long-running entity event processing
* GPLA-4770: Fixed Box feed authentication errors caused by expired refresh tokens


# December 11: Support for GPT-5.2, Attio CRM, mirror sync mode, filter modes

### New Features

* 💡 Graphlit now supports **OpenAI GPT-5.2** with the model enum `GPT52_400K`.
* 💡 We have completed **Attio CRM integration** as a data source, allowing you to ingest companies, contacts, and custom CRM objects directly into your knowledge graph. Custom object mappings let you model your specific CRM structure (LPs, Investments, External Funders, etc.).
* We have added **MIRROR sync mode** for Azure blob and S3 feeds, enabling bidirectional synchronization where deletions in the source are reflected in Graphlit.
* We have added **filter modes** (`any`, `all`, `only`) for collections and other filter objects, providing more precise control over how multiple filter values are combined.
* We have added an **entity relationship lookup query** for efficient traversal of knowledge graph relationships without scanning the full graph.
* **MCP server support** is now available directly from Graphlit projects, similar to Zine. Developers can expose their project content via remote MCP with API key authentication.
* We have added a **relevance threshold** property to filter objects, allowing developers to override the default vector similarity threshold.

### Bugs Fixed

* GPLA-4758: Fixed precondition failures when updating content concurrently
* GPLA-4764: Person entities now correctly generate vector embeddings
* GPLA-4763: Fixed specification upsert matching similar names incorrectly (e.g., "GPT-5" vs "GPT-5.1")
* GPLA-4724: Fixed slow CosmosDB queries with multi-partition fallback retries
* GPLA-4682: Credits quota exceeded now returns proper `PaymentRequired` status instead of `BadRequest`


# November 28: Support for entity resolution, agentic GraphRAG, user-scoped knowledge graphs

### New Features

* 💡 Graphlit now supports **LLM-based entity resolution** via the `resolveEntities` mutation. You can provide a list of JSON-LD entities to the model, and it will return a resolved list with fields merged or new entities split out, helping to deduplicate and consolidate your knowledge graph.
* 💡 We have expanded our **agentic GraphRAG capabilities** by breaking out specific SDK calls for graph tools, allowing an agentic LLM to manipulate the knowledge graph directly rather than relying solely on graph strategy configuration.
* Graphlit now supports **user-specific knowledge graphs**. For multi-tenant applications, each user can have their own isolated knowledge graph rather than sharing a project-wide graph.
* We have added support for **on-demand entity enrichment**. You can now select entities and enrich them with external data sources like Diffbot, Crunchbase, or Parallel to fill in websites, addresses, funding information, and more.
* Observable entities now expose their `feed` property to the API, and you can filter entities by feed using the updated observable filter.

### Bugs Fixed

* GPLA-4748: Fixed entity validation errors when upserting Organization entities
* GPLA-4743: Enrichment now correctly creates and assigns Founder relationships from JSON-LD
* GPLA-4725: `graph` query now returns inter-entity edges that were being ignored
* GPLA-4728: Deleting a CRM feed now recursively deletes entities sourced from it


# November 17: Support for entity feeds, CRM integration, research feeds, GPT-5.1

### New Features

* 💡 Graphlit now supports **entity feeds** for direct knowledge graph population. Using the Parallel "Find All" API, you can discover and ingest relevant entities directly into your knowledge graph without requiring associated content.
* 💡 We have added a **Research feed** type that kicks off async deep research tasks via the Parallel Task API and ingests the resulting Markdown as new content. This enables automated research workflows that populate your project with synthesized information.
* We have added support for **GPT-5.1** with the model enum `GPT51_400K`.
* We have extended **Attio CRM support** with custom object mapping, allowing you to ingest companies, contacts, and custom objects from your CRM directly into the knowledge graph.
* The `graph` query now provides a direct way to query the knowledge graph exclusive of content, returning nodes and edges without filtering through content.
* **Azure blob metadata** is now ingested and stored in content, making custom blob properties available for retrieval.
* **GitHub commits** now return file changes and diffs via the Octokit integration.

### Bugs Fixed

* GPLA-4701: Reddit posts now correctly return Markdown content
* GPLA-4694: Migrated to latest Jira JQL API after deprecation of previous endpoint
* GPLA-4709: Fixed text-to-audio publishing errors in alert workflows
* GPLA-4708: Fixed Sora video publishing size validation errors
* GPLA-4707: Feed identifiers can now be assigned via `updateFeed`


# November 7: Support for commit and PR feeds, Parallel web search, contact list feeds

### New Features

* 💡 Graphlit now supports **GitHub commit and pull request feeds**, allowing you to ingest commits and PRs similar to how GitHub issues are handled. Commit metadata includes file changes and diffs.
* 💡 We have added the **Parallel (p0) web search API** as a new search service type via `PARALLEL` in `searchWeb`. Similar to Exa and Tavily, this provides an additional option for web search and research.
* We have added support for **contact list feeds** to pull Person entities from Google, Microsoft, and other contact providers directly into your knowledge graph.
* Organization owners can now **create, delete, and upgrade projects via the GraphQL API**, enabling programmatic project management without using the portal.
* We have added support for **Gemini 3 Pro Preview** with the model enum `GEMINI_3_PRO_PREVIEW`.
* Content summarization prompts are now **customized by content type**, providing better descriptions for commits, PRs, code, and other specialized content.

### Bugs Fixed

* GPLA-4717: Fixed MIME type inference errors for certain web pages
* GPLA-4715: Fixed OpenRouter API response parsing for Claude models
* GPLA-4684: Google Drive folder ingestion now correctly assigns relative folder paths


# October 24: Support for content versioning, duplicate policies, AskGraphlit MCP server

### New Features

* 💡 Graphlit now supports **content version control** when restarting content in-place. When documents are re-ingested from connected feeds, version history is automatically tracked, enabling developers to audit document revisions over time.
* 💡 We have added configurable **duplicate handling policies**. You can now specify whether duplicates are allowed across feeds or across the entire project. An event is fired when duplicates are detected, enabling custom handling workflows.
* We have launched **AskGraphlit**, a free public MCP server that incorporates our latest documentation. Developers can use this with AI assistants like Claude to get help coding with Graphlit SDKs.
* Content metadata is now more flexible: you can update `creationDate` and `modifiedDate` on content items, and filter contents by `modifiedDate` ranges.
* Child content (extracted images, meeting recordings, package contents) is now automatically added to the parent's collection when using collection workflows, simplifying organization for complex document hierarchies.

### Bugs Fixed

* GPLA-4657: Fixed search clause limit errors when filtering by large numbers of feeds or content IDs
* GPLA-4672: Optimized default model selection for entity extraction to reduce costs


# October 17: Support for video publishing, embedding opt-out, Claude Haiku 4.5

### New Features

* 💡 Graphlit now supports **Google Veo 3 video publishing**, extending our content generation capabilities beyond audio (ElevenLabs) to include AI-generated video.
* Developers can now **opt out of vector embeddings** during enrichment via storage policy configuration. This enables keyword-only search for specific content types without the overhead of vector indexing.
* We have added support for **message collections** for Slack, Teams, and Discord feeds via `enableMessageCollections` in `IngestionWorkflowStage` configuration, mirroring the existing `enableEmailCollections` functionality for threaded conversation organization.
* We have added support for **Claude Haiku 4.5** with the model enum `CLAUDE_4_5_HAIKU`.
* We have added **Exa Code** as a new web search option via `EXA_CODE` search service type, optimized for developer-focused code and documentation queries.

### Bugs Fixed

* GPLA-4661: Fixed race condition where content ingested near alert window boundaries could be missed
* GPLA-4660: Fixed resource exhaustion error when exporting large MP4 files from Google Drive
* GPLA-4648: Alert webhook URIs can now be updated via `updateAlert` without recreating the alert
* GPLA-4652: Fixed Anthropic Claude error when both `temperature` and `top_p` were specified
* GPLA-4643: Workflow actions now properly update via `upsertWorkflow` mutation


# October 7: Support for GitHub repository listing, child content in RAG, feed improvements

### New Features

* 💡 Graphlit now supports **listing available GitHub repositories** based on an authenticated connector, making it easier to discover and configure repository feeds programmatically.
* 💡 LLM conversations now **recursively include child content** (extracted images, attachments) in the context provided to the model. This ensures AI responses can reference all related content from a parent document without manually adding children to collections.
* We have added **last modified user tracking** for Google Drive and SharePoint feeds. The `modifiedBy` field now captures who made changes to documents.
* Reddit post ingestion now extracts **full comment threads with author and timestamp metadata** into structured `PostMetadata`, separate from the rendered Markdown.
* We have added support for **gpt-image-1-mini** for OpenAI image generation.

### Bugs Fixed

* GPLA-4650: Fixed image search via LLM for GPT-5 and Claude 4.5 Sonnet models
* GPLA-4631: Fixed alerts not firing on in-place Google Drive document updates
* GPLA-4630: Fixed missing line breaks in Markdown conversion from PDFs
* GPLA-4640: Fixed duplicate detection not recognizing modified Google Docs across feeds
* GPLA-4621: Resolved feed query failures for previously imported feeds


# September 30: Support for ingestion performance, search stability, and token accounting accuracy

### New Features

* :bulb: This release includes broad **performance optimizations** across ingestion, search, and summarization pipelines. Developers should see faster indexing for large document batches and improved throughput when embedding concurrent content streams.
* We’ve also refined **token accounting and reporting** within the billing subsystem, ensuring real-time accuracy for both input and output token usage across all connected model providers.

### Bugs Fixed

* GPLA-4590: Resolved high-concurrency queue deadlocks affecting ingestion of large feed batches
* GPLA-4594: Improved summarization retry logic for timeouts on long transcripts
* GPLA-4601: Corrected mis-reported embedding-cost metrics for Gemini models


# September 26: Support for regex path filters, token thresholds, and feed indexing updates

### New Features

* :bulb: Graphlit introduces a **minimum-token threshold** for extraction, ensuring small or incomplete snippets are excluded from embedding and search operations.
* We’ve also added **regex-based include / exclude path filters** for site feeds, giving developers more control over which pages are crawled and indexed.
* After refreshing environment JWT secrets, the Portal now automatically reloads connected data APIs, eliminating the need for manual cache resets.

### Bugs Fixed

* GPLA-4554: Fixed parsing of email items into feed posts across multi-part messages
* GPLA-4018: Removed erroneous `402` responses when querying usage logs
* GPLA-4560: Resolved RSS feed parsing failures with malformed XML payloads
* GPLA-4565: Restored Slack user lookup when bot tokens lacked the proper `users:read` scope
* GPLA-4571: Fixed Notion item parsing inconsistencies affecting multi-block documents


# September 14: Support for folder-scoped collections, Voyage embeddings, and Portal improvements

### New Features

* :bulb: Graphlit now supports **folder-scoped collections**, enabling developers to organize content hierarchically for more granular access control and retrieval.
* We’ve integrated the latest **Voyage 3.5 embedding** and **reranking** models, providing improved semantic accuracy and retrieval precision.
* Alerts now support **saved views**, allowing filtered alert dashboards that persist across sessions. 
* We also introduced several quality-of-life updates in the Portal UI, including better context indicators and pagination performance. Additionally, the payment workflow now supports **Stripe 3D Secure**, ensuring compliance with modern authentication standards.

### Bugs Fixed

* GPLA-4497: Fixed caching of archive-site SAS tokens used in content replication
* GPLA-4485: Improved reliability when querying Azure Log Analytics workspaces
* GPLA-4487: Resolved intermittent embedding-vector generation errors
* GPLA-4531: Clarified handling of embedding failures as hard errors to prevent silent retries
* GPLA-4528: Addressed Jina image-embedding failures in high-throughput pipelines


# August 31: Support for Google 'nano-banana' image generation, email filtering options.

### New Features

* :bulb: Graphlit now offers improved email-list handling for Gmail and Outlook integrations, reducing duplicate processing and improving attribution of threaded messages.
* Support has been added for Google’s experimental **“nano banana”** image-generation model. This extends publishing capabilities to additional diffusion back-ends.
* We also introduced configurable **email-filtering controls** for ingestion feeds, allowing developers to target messages by subject, sender, or folder path.

### Bugs Fixed

* GPLA-4495: Resolved ingestion failures when processing YouTube video content, ensuring proper transcript extraction and thumbnail handling


# August 25: Support for custom AWS Bedrock endpoints, feed-specific backup, bug fixes

### New Features

* :bulb: Graphlit now supports specifying `endpoint` and `region` properties when registering **AWS Bedrock** models. This gives developers finer control over latency and compliance by targeting preferred AWS regions.
* Backup operations can now be **filtered by feed**, enabling partial exports or restores for specific data sources without performing full-tenant backups.

### Bugs Fixed

* GPLA-4465: Improved Wikipedia enrichment reliability for entity linking and metadata extraction
* GPLA-4451: Microsoft email and calendar feeds now fail gracefully on invalid tokens
* GPLA-4449: Resolved intermittent Gmail message listing errors during ingestion
* GPLA-4474: Fixed rare Google prompt-completion errors that occurred during feed indexing


# August 12: Support for OpenAI GPT-5, Anthropic Opus 4.1 models, and CLI restore

### New Features

* :bulb: Graphlit now supports the latest OpenAI GPT-5 and Anthropic Opus 4.1, and Opus 4.1 Mini model variants. Developers can reference them through the `GPT5_400K`, `OPUS_4_1`, and `OPUS_4_1_MINI` enums when creating or updating model configurations.
* We’ve enhanced the Graphlit CLI with a new **restore** command, allowing you to rehydrate backed-up feeds or collections directly from the command line. This simplifies environment migrations and local testing workflows.

### Bugs Fixed

* GPLA-4429: Resolved ingestion performance issues for extremely large memory objects
* GPLA-4434: Improved ingestion stability when content metadata was malformed
* GPLA-4439: Corrected authentication mapping for GitHub feed properties


# July 17: Support for xAI Grok models, YouTube audio download, bug fixes

### New Features

* :bulb: Graphlit now supports xAI as a model provider, with the latest Grok 3 and Grok 4 models.
* We have added support for the latest Groq and Cerebras models, including Kimi K2 and Qwen 3.
* We have added support for downloading YouTube audio tracks from a YouTube video, when using `ingestUri` mutation.

### Bugs Fixed

* GPLA-4370: Unable to update feed read limit after creation
* GPLA-4343: Split paragraphs with long lines is too slow
* GPLA-4401: Update model cards to have ALL\_CAPS enums
* GPLA-4412: Failed to stream response with Cohere Command A


# July 4: Support for Google and Microsoft calendar feeds, saved views, bug fixes

### New Features

* :bulb: Graphlit now supports calendar feeds from Google and Microsoft. Each calendar event will be ingested as an `EVENT` content type, and will parse available metadata.
* :bulb: Graphlit now supports the creation of `views`, as saved filters which can be used to retrieve contents.
* :bulb: Graphlit now supports the creation of `connectors`, associated with `user` objects, for storing the configuration of integrations like Slack, or authentication services, like Google or Microsoft.
* We have added support for assigning an external `identifier` with all the ingestion mutations, such as `ingestText` and `ingestUri`.  Now you can assign your own system's identifier to any content object, and query contents by that external identifier.
* We have added query methods to enumerate drives and folders for OneDrive, Google Drive, Dropbox and Box.
* We have added a query method for enumerating Discord channels.

### Bugs Fixed

* GPLA-4358: Failed to summarize contents during publishing
* GPLA-4370: Unable to update feed read limit after creation


# June 20: Support for LLM streaming in Typescript SDK, bug fixes

### New Features

* :bulb: Graphlit now supports LLM streaming natively in our Typescript SDK.  Via `streamAgent` method in the SDK, you can prompt any LLM provider and receive a stream of token and tool calling events, which can be rendered into your application.  Also, we support `promptAgent` for a similar DX with non-streaming support.
* We have added support for 'pull quotes' as a summarization type. You can have the LLM extract relevant, publishable quotes from any text or transcript.

### Bugs Fixed

* GPLA-4327: formatConversation not adhering to 'None' search type
* GPLA-4340: Twitter messages should have default name
* GPLA-4334: Losing line breaks in markdown from ImageMezzanine


# June 10: Support for Reducto PDF preparation, text geotagging, bug fixes

### New Features

* :bulb: Graphlit now supports [Reducto](https://reducto.ai/) as a preparation service for PDF text extraction.
* We have added support for geotagging as a summarization type. We will use an LLM to infer one or more geolocated features from the text or transcript, and return the data in GeoJSON format.
* ⚠️ We have removed support for Azure OpenAI built-in model deployments.  You can still provide your own custom deployments of Azure OpenAI models, from your own Azure subscription, but we no longer support GPT-4 and other legacy models with Azure OpenAI.

### Bugs Fixed

* GPLA-4306: Uppy handler not parsing 'userId' from JWT
* GPLA-3720: Failing to process Spreadsheet leaves content in Indexed state.
* GPLA-4316: Failing to filter content with empty string on 'boundaries'
* GPLA-4279: Google Drive feed still says 'Running' but it's read all 100 files


# May 31: Support for Anthropic Sonnet & Opus 4 models, ingestEvent mutation, bug fixes

### New Features

* :bulb: Graphlit now supports Anthropic Sonnet 4 and Opus 4 models.
* We have added the `ingestEvent` mutation to the API, so you can create `EVENT` content to track in-person or virtual events.
* We have added support for Amazon Bedrock models, which are charged as Graphlit credits. Previously we had supported BYOK (bring your own key), but now you can call any Bedrock model and charge it to your project usage.

### Bugs Fixed

* GPLA-4253: Not passing tool calls/responses with previous conversation messages
* GPLA-4263: Publishing shouldn't use default project specification
* GPLA-4262: Losing line breaks in Markdown copied from web page


# May 11: Support for Amazon Bedrock models, McPoogle MCP search engine, bug fixes

### New Features

* :bulb: Graphlit now supports Amazon Bedrock models for LLM completion, preparation and extraction. Currently, we only support 'bring your own key' for models in your Bedrock environment, but will support per-credit usage of Bedrock models in an upcoming release.
* We have launched our MCP search engine, called [McPoogle](https://www.mcpoogle.com), for finding MCP servers and tools. We also launched McPoogle as a [remote MCP server](https://github.com/graphlit/mcpoogle-mcp-server-remote), which you can use within AI-native IDEs such as Cline, Cursor or Windsurf.
* We have launched a [remote MCP server](https://github.com/graphlit/ask-graphlit-mcp-server-remote) for 'Ask Graphlit', where you can get code generation of Graphlit SDK samples within AI-native IDEs such as Cline, Cursor or Windsurf.
* We have added support for providing a 'seed' image when performing image generation. You can select an example image content, by reference, and use that to guide the image generation model for following your user prompt.

### Bugs Fixed

* GPLA-4194: Parsing raw GitHub README has extra HTML at end
* GPLA-4202: Round-trip markdown to TextDocument loses Markdown list items
* GPLA-4235: Hybrid RRF threshold is removing too many sources
* GPLA-4233: Can't send just image & system prompt to Anthropic LLM


# April 26: Support for OpenAI image generation, GPT-4.1, o3 and o4-mini models, bug fixes

### New Features

* :bulb: Graphlit now supports the latest [OpenAI image generation mode](https://platform.openai.com/docs/guides/image-generation?image-generation-model=gpt-image-1)l. As part of the publishing functionality, you can now publish contents to an image format, similar to how we already support publishing to ElevenLabs audio formats.
* We have added support for the Gemini 2.5 Flash Preview model, with the `GEMINI_2_5_FLASH_PREVIEW` model enum.
* We have added support for the OpenAI [GPT 4.1 ](https://platform.openai.com/docs/models/gpt-4.1)(full), [Mini](https://platform.openai.com/docs/models/gpt-4.1-mini), and [Nano](https://platform.openai.com/docs/models/gpt-4.1-nano) models, with model enums `GPT41_1024K`, `GPT41_MINI_1024K`, `GPT41_NANO_1024K`.
* We have added support for OpenAI o3 and o4-mini models, with the model enums `O3_200K`, and `O4_MINI_200K`.
* We have added support for injecting an image content source into the RAG conversation. Previously, you weren't able to upload an image, and then start a RAG conversation which said "Describe this image". Now this will reference the recently upload image properly.
* We have added a new field called 'similarContents' which provides the legacy similarity search by contents. The existing 'contents' field will now provide filtering by a list of contents.
* :zap: We have made a change to the semantics of the 'contents' field in the ContentFilter object. When this was originally implemented, we'd retrieve contents similar to the vector embeddings of those provided in the 'contents' field. However, the other fields like 'collections', 'feeds', etc. do not use similarity search, which has been confusing for many.

### Bugs Fixed

* GPLA-4075: Failed to parse large email payload.  (Now truncating to 917504 characters.)
* GPLA-3865: Wasn't capping screenshot height on HTML image extraction.
* GPLA-4161: AskGraphlit: Unable to format conversation; exceeding token budget.


# April 13: Support for memory, email thread collections, Groq Llama 4 models, bug fixes

### New Features

* :bulb: Graphlit now supports `MEMORY` as a new content type.  Via the `ingestMemory` mutation, you can create a textual memory for your user or agent. (Not supported at project scope.)  Memories will automatically have entity extraction applied to create an 'interest graph' from the accumulated memories. Memories will be included with retrieved sources for RAG conversations, via vector similarity. (Coming soon: memories will support time-to-live (TTL), so they will age out automatically.)
* We have added support for auto-creating collections for email threads. You can opt into this behavior with `enableEmailCollections` in `IngestionWorkflowStage` configuration.
* We have added support for Google Drive service account authentication. You can now provide the JSON credentials from Google Cloud Console to authenticate to a Google Drive feed.
* We have added support for the latest Llama 4 models in the Groq provider, with the model enums: `LLAMA_4_MAVERICK_17B`, `LLAMA_4_SCOUT_17B`.
* We have added `threadId` to email metadata, for tracking the original email thread identifier.

### Bugs Fixed

* GPLA-4103: Unable to lookup content by original URI
* GPLA-4109: Not parsing out hyperlinks from HTML list items


# March 27: Support for Twitter/X feed, Gemini 2.5 Pro model

### New Features

* :bulb: Graphlit now supports Twitter/X as a data feed. You can ingest the posts or mentions by Twitter username, including attachments, or search for recent posts (in last 7 days). You will need to provide an OAuth2 bearer token generated from the Twitter Developer Portal for your paid access to their API.
* Graphlit now supports the Google Gemini 2.5 Pro Experimental model, with the model enum `GEMINI_2_5_PRO_EXPERIMENTAL`.
* We have added `formats`and `fileExtensions`as filterable properties in the ingestion stage of the workflow.
* We have increased the number of SharePoint folders returned from `gettSharePointFolders`to 5000.


# March 15: Support for Podscan search, image similarity search, 'exists' and 'upsert' mutations

### New Features

* :bulb: Graphlit now supports podcast searches using the [Podscan search API](https://podscan.fm/) with the `searchWeb`mutation.  In addition to Tavily and Exa, you can now specify `PODSCAN`as a search service type. It will return the URL to the downloadable audio track of the podcast episode.
* :bulb: Graphlit now supports the ingestion of podcast episodes with the web search feed. We will automatically ingest and transcribe the resulting podcast episodes into the Graphlit project, which become available for retrieval and RAG conversations.
* Graphlit now supports direct similarity search based on image embeddings, by providing Base64-encoded `imageData`and `imageMimeType`properties in the `ContentFilter`object. We will use your configured image embeddings model, such as Jina, to embed the provided image and then use for vector search to return similar contents.
* Graphlit now supports `workflowExists`, `specificationExists`and `feedExists`queries, which returns whether any workflow, specification or feed exists that matches the provided filter criteria.
* We have added upsert mutations for specifications (`upsertSpecification)` and workflows (`upsertWorkflow`), to simplify the creation of specifications and workflows, when you're not sure if they already exist.  If the entity already exists by name, it will update the existing entity, else it will create a new entity with that name.
* We have added upsert mutations for labels (`upsertLabel`) and categories (`upsertCategory`), to simplify the creation of labels and categories by name, when you're not sure if they already exist.
* :zap: We have changed the default number of retrieved entities to 100 (used to be 1000), to increase retrieval performance. You can assign the `limit`property on filter objects to raise this default.


# March 13: Support for classification workflow, notifications, Cohere Command A model, bug fixes

### New Features

* :bulb: Graphlit now supports a **classification** stage in the workflow object. As content is being ingested, and after the extraction stage in the workflow, you can optionally enable classification rules to apply labels to the content based on regex patterns or LLM prompts.  For example, an LLM prompt classification rule can use Gemini 2.0 Flash to analyze the Markdown text of an email, and assign the 'Invoice' label to the content automatically, if the prompt determines the email contains an invoice to be paid.
* Graphlit now supports the [Cohere Command A model](https://cohere.com/blog/command-a), with the model enum `COMMAND_A`.
* We have added notification mutations to send textual notifications to Slack, email or webhook.  The new `sendNotification`mutation is used in MCP tools to directly send the Markdown output from a tool to Slack, email or webhook.

### Bugs Fixed

* GPLA-3961: Failed to extract hyperlinks from Word document
* GPLA-3946: Google Drive feed is ingesting a duplicate rather than in-place
* GPLA-3905: Slow to load JSON mezzanine for CSV
* GPLA-3966: If mapWeb doesn't find anything, just return original URL
* GPLA-3959: Failed to process hyperlink spreadsheet cell


# March 6: Support for MCP Server, Mistral OCR, retrieveSources, GPT-4.5, Sonnet 3.7, bug fixes

### New Features

* :bulb: Graphlit now supports the Anthropic MCP (Model Context Protocol) standard, through our new open source MCP server.  [Read more in our blog](https://www.graphlit.com/blog/graphlit-mcp-server).
* Graphlit now supports the `retrieveSources`mutation in the API, which provides reranked retrieval of content sources, for use in agents and MCP clients, outside of RAG conversations. We have also added support for filtering retrieved sources by feed(s) and collection(s).
* Graphlit now supports the [Mistral OCR model](https://docs.mistral.ai/capabilities/document/) for document preparation, with the preparation service enum of `MISTRAL_DOCUMENT`.  When selecting this document preparation service, it will use the `mistral-ocr-latest`model with PDFs. (Note: There are known issues with the model extracting embedded images without parsing text from them.)
* We have added support for OpenAI GPT-4.5 model with the model enum `GPT45_PREVIEW_128K`.
* We have added support for the Anthropic Claude 3.7 model with the model enum `CLAUDE_3_7_SONNET`, including the [thinking token budget.](https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking)
* When uploading files via the Graphlit Tus server, we now support the assignment of `workflowId`and `collectionId`, so that the ingested file can automatically be assigned to a collection, and have the specified workflow applied.
* We have added the `linearProjects`query to the API, for enumerating Linear projects in your authenticated account.

### Bugs Fixed

* GPLA-3912: Can't upload same filename twice via Uppy
* GPLA-3892: Failed to process table with Azure Doc Intelligence
* GPLA-3869: Failed to get Microsoft email delta
* GPLA-3947: Failed to parse hyperlinks from Word doc
* GPLA-3878: PDF extraction can be more detailed on charts
* GPLA-3830: Failed to get Google Drive URI (invalid chars in filename)
* GPLA-3908: Not adding line break with Sonnet markdown extraction


# February 16: Support for Trello feed, Assembly.AI audio transcription, OpenAI o3-mini, bug fixes

### New Features

* :bulb: Graphlit now supports Trello feeds for ingesting Trello cards and/or cards from Trello boards. Trello feeds require your Trello API key and token for authentication.  You can provide either a list of Trello card identifiers or Trello board identifiers for ingestion as `ISSUE`content type. ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2025_02_19_Analyze_Trello_Feed_Ingestion_Costs_%26_Usage.ipynb))
* :bulb: Graphlit now supports Assembly.AI audio transcription, as an optional service type configuration (`ASSEMBLY_AI`) in the preparation workflow.  You can select the Assembly.AI properties of `enableRedaction`, `enableSpeakerDiarization`, `detectLanguage`and `language`. ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2025_02_19_Transcribe_Podcast_using_Assembly_AI.ipynb))
* We have added support for OpenAI [o3-mini](https://platform.openai.com/docs/models#o3-mini) model with the model enum `O3_MINI_200K`.
* We have added a `reasoningLevel`property to OpenAI model properties, with potential values of `HIGH`, `MEDIUM`, and `LOW`.
* We have added support for the latest Deepgram [Nova-3](https://developers.deepgram.com/docs/models-languages-overview#nova-3) model with the model enum `NOVA3`.

### Bugs Fixed

* GPLA-3793: Not stopping ingestion if validate tools fails
* GPLA-3807: Not handling HTML links w/o URI scheme
* GPLA-3827: Handle transient error on getting Microsoft email folder
* GPLA-3819: Handle missing Crunchbase entities without erroring observable


# January 30: Support for Uppy file uploader, Deepseek Reasoner model, bug fixes

### New Features

* :bulb: Graphlit now supports the [Uppy](https://uppy.io/) file uploader with an integrated Tus server.  By configuring Uppy to use the new Graphlit Tus server, and passing the JWT for your Graphlit project, you can quickly integrate user file uploads into your application. ([Github Next.js Example](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/uppy))
* Graphlit now supports the [Deepseek Reasoner](https://api-docs.deepseek.com/quick_start/pricing) model with the model enum `REASONER`.
* Graphlit now supports the Groq Deepseek R1 (Llama 70b Distill) model with the model enum `DEEPSEEK_R1_LLAMA_70B_PREVIEW`.

### Bugs Fixed

* GPLA-3770: Failed to parse missing Gmail message
* GPLA-3774: Failed to create workflow with by-name observation


# January 19: Support for at-cost LLM token pricing, multi-tenant feed deletion, bug fixes

### New Features

* :bulb: Graphlit now charges for LLM tokens at the cost of the LLM provider. For example, if you use 10,000 input tokens and 1000 output tokens from OpenAI GPT-4o, Graphlit will convert the token usage to credits at the cost of the LLM provider, without any additional margin added.
* :bulb: Graphlit now charges a flat-rate of 2.5 credits ($0.25/Mtok on Hobby tier) per million input tokens for LLM completions, no matter what LLM is used.  This will make the cost of LLM usage cheaper in the Graphlit platform, and more consistent, no matter if you are using OpenAI o1 or Deepseek Coder.  **This ends up reducing the cost of token-intensive operations by 25% or more, on average.**
* Graphlit now supports deleting multi-tenant entities from project-scope. For example, if an end-user has created a feed with a unique 'owner ID', you can now call deleteFeed from project scope to delete the user's feed. 
* We now support the retrieval of multi-tenant conversations from project-scope. For example, if an end-user has created a conversation with a unique 'owner ID', when you query conversations from project scope, the end-user's conversations will be included with the retrieved conversations.

### Bugs Fixed

* GPLA-3764: Groq custom model not using proper API endpoint
* GPLA-3754: Not handling Microsoft emails with null parent folder ID
* GPLA-3747: Need retry on Jina embeddings
* GPLA-2962: Not crawling website where host (w/o 'www') doesn't match sitemap host
* GPLA-3750: Not crawling website when sitemap returns application/rss+xml MIME type
* GPLA-3742: DeleteContents/DeleteAllContents have missing arguments in Node.js SDK
* GPLA-3757: Error on indexing of PDF was not assigning content.error field.


# January 10: Support for conversation message images, email filtering, Diffbot API key, bug fixes

### New Features

* :bulb: Graphlit now supports sending Base64-encoded images with user messages to the `prompt`mutation. You can provide the user `prompt`, with an optional `mimeType`and `data`property to have the LLM prompt use the image in its prompt completion.  This applies to the conversation `messages`parameter as well. Also, now the user `prompt`parameter is optional, and you are able to just provide the conversation messages array only, if desired. 
* Graphlit now supports email filtering for Microsoft and Google email. We have added flags in the email feed properties for `includeSpam`, `includeDeletedItems`, and `excludeSentItems`which all default to False. We have also added `inboxOnly`which you can set to True to only read emails from the Inbox folder (applies to reading Past emails; reading new emails already reads just from the Inbox folder).
* We have added a `disableInheritance`flag into the `ContentFilter`object, which will disable the default inheritance of content from project-scope to tenant-scope.  By setting this to True, and querying contents within a tenant, you will only get back tenant contents, and nothing from the parent project.
* We have added support for bringing your own Diffbot API key.  You can assign `key`to the `diffbot`property in the `EntityEnrichmentConnectorInput`object, when creating your workflow

### Bugs Fixed

* GPLA-3723: Failing to ingest DOCX from Google Drive, with multitenancy
* GPLA-3706: Handle Microsoft Graph resync required properly


# January 4: Support for askGraphlit mutation, storage policies, bug fixes

### New Features

* :bulb: Graphlit now supports the `askGraphlit`mutation, which is a built-in support chat which lets you create SDK code samples, or ask questions about the Graphlit Platform itself.  It is similar to `promptConversation`but is prefilled with relevant content, and doesn't rely on the existing content in the project. ([Next.js Sample Application](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/ask)) ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2025_01_05_Ask_Graphlit.ipynb))
* Graphlit now supports the `storage`stage in the `workflow`object, which allows you to control the storage policies and whether duplicate contents (by eTag or URI) are allowed.
* Graphlit now supports storage policies with the `StoragePolicyTypes`enum, which instructs the platform to `ARCHIVE`the content indefinitely (the current default), or `MINIMIZE`the storage footprint by removing the cached master file when the workflow finishes.  With `MINIMIZE`, the `masterUri`property will be null after the workflow finishes. We will be adding a storage policy for time-based aging (deletion) of content in the future.
* We have added support for the `includeDetails`flag in the `publishContents`mutation, so you can get more detailed information on the intermediate results during content publishing.
* We have added support for the Google Flash 2.0 Experimental Thinking model, with the enum `GEMINI_2_0_FLASH_THINKING_EXPERIMENTAL`.  Note, the experimental models have no charge, and the 'Thinking' model only supports 32K token context window.
* :zap: We have changed the parameter name in `sharePointFolders`from `driveId` to `libraryId`for consistency.

### Bugs Fixed

* GPLA-3671: Failed to index PDF from Dropbox download link
* GPLA-3675: Incorrectly inferring HTML PDF page wrapper as PDF


# December 27: Support for LLM fallbacks, native Google Docs formats, website unblocking, bug fixes

### New Features

* :bulb: Graphlit now supports LLM fallbacks which can help protect your application from model provider downtime.  By assigning the `fallbacks`property when creating your conversation, you can provide an optional list of LLM specifications to be used (in order).  These fallback specifications will only be used when we failed to prompt the conversation via the main specification.  Caveat, the RAG pipeline will only use the strategies provided in the main specification for prompt rewriting, content retrieval, etc.  Content is not re-retrieved upon fallback - the formatted LLM prompt will be tried against each fallback specification in succession until one succeeds. ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2024_12_28_Configure_LLM_fallbacks_between_OpenAI%2C_Groq%2C_and_Anthropic.ipynb))
* :bulb: Graphlit now supports querying of all available models, through the new `models`query in the API.  This returns the model enum, model service type enum, description, and several other useful details about the models. ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2025_01_05_Query_Model_Cards.ipynb))
* Graphlit now supports the ingestion of native Google Docs, Google Sheets and Google Slides documents from Google Drive feeds.  These formats will be auto-exported to the corresponding Microsoft Office format (DOCX, XLSX, PPTX) prior to ingesting as content.
* Graphlit now supports unblocking of websites, such as those using Cloudflare.  You can set `enableUnblockedCapture`to true on the `PreparationWorkflowStage`to enable unblocking - through our integration with [Browserless.io ](https://www.browserless.io/)headless browser service.  This does incur an additional cost per page, compared to normal web page ingestion.
* We have added support for assigning observations to contents ingested via feeds.  By assigning `observations`to the `IngestionWorkflowStage`in workflow object, you can assign Labels, Organizations, etc. without needing to use entity extraction.
* We have added support for assigning observations when ingesting content via `ingestUri`, `ingestText`, etc. mutations. By passing `observations`as a parameter, similar to \`collections\`, you can assign Labels, Organizations, etc. without needing to use entity extraction.
* :zap: We have changed the response type of the `publishContents`mutation to return `PublishContents`type.  This new `PublishContents`type wraps the published `Content`object, and includes the new `Details`property of `PublishingDetails`type. We have added an `includeDetails`parameter to `publishContents`mutation, which will fill in the Details property with a list of intermediate content summaries and the published text, among other publishing metrics. 
* :zap: We have changed the behavior of `publishContents`such that, if no content was retrieved for publishing, the mutation returns a null content object rather than returning an error.

### Bugs Fixed

* GPLA-3645: Table headers merged together on web scrape
* GPLA-3634: Failed to extract pages from PDF with empty hyperlink text
* GPLA-3633: Not handling empty observables properly for reranking


# December 22: Support for Dropbox, Box, Intercom and Zendesk feeds, OpenAI o1, Gemini 2.0, bug fixes

### New Features

* :bulb: Graphlit now supports Dropbox feeds for ingesting files on the Dropbox cloud service. Dropbox feeds require your `appKey`, `appSecret`, `redirectUri`and `refreshToken`to be assigned. The feed also accepts an optional `path`parameter to read files from a specific Dropbox folder.
* :bulb: Graphlit now supports Box feeds for ingesting files on the Box cloud service. Box feeds require your `clientId`, `clientSecret`, `redirectUri`and `refreshToken`to be assigned.
* :bulb: Graphlit now supports Intercom feeds for ingesting Intercom Articles and Tickets. We will ingest Intercom Articles as `PAGE`content type, and Tickets as `ISSUE`content type. Intercom feeds require the `accessToken`property to be assigned.
* :bulb: Graphlit now supports Zendesk feeds for ingesting Zendesk Articles and Tickets.  We will ingest Zendesk Articles as `PAGE`content type, and Tickets as `ISSUE`content type. Zendesk feeds require the `accessToken`property and your Zendesk subdomain to be assigned.
* Graphlit now supports the latest OpenAI o1 model, with the model enums `O1_200k`and `O1_200k_20241217`.
* Graphlit now supports the latest Gemini Flash 2.0 Experimental model, with the model enum `GEMINI_2_0_FLASH_EXPERIMENTAL`.
* Graphlit now supports the latest Cohere R7B model, with the model enum `COMMAND_R7B_202412`.
* Graphlit now supports returning the low-level details from prompting RAG conversations, by adding the `includeDetails`parameter and setting to True. This includes details on the number of sources, the exact list of messages provided to the LLM, and more.
* We have added support for filtering of observables, such as Person or Organization, by URI property.
* We have added the ability to bypass semantic search in content retrieval with conversations. You can assign `NONE`for the conversation search type, and it will ignore the user prompt when retrieving content.  It will inject all contents resulting from the content filter into the RAG prompt context.
* We have added a new `createdInLast`property to all entity filters, which allows easier filtering of entities created within a recent time period. Also, we have added a new `inLast`property to the content filter, which allows easier filtering of content authored within a recent time period. For example, find all images taken in the last 3 days, or find me all emails I received yesterday.
* We have added support for the latest Azure AI Document Intelligence models, with enums `US_PAY_STUB`, `US_BANK_STATEMENT`, and `US_BANK_CHECK.`
* We have added support for Google Drive and OneDrive feeds to ingest specific files by providing a list of file identifiers (`files`), in addition to the folder identifier (`folderId`).  If files identifiers are provided, they take precedence over the folder identifier.
* :zap: For projects upgraded to the Starter Tier after Dec 9, 2024, we have removed the content items limit. Now you can store an unlimited number of content items (i.e. files, web pages, Slack messages) on the Starter or Growth Tiers.  If you have an existing project on the Starter Tier, please reach out and we will manually remove that content item limit on the project. 

### Bugs Fixed

* GPLA-3529: Can't assign collection to multitenant content
* GPLA-3579: Should decode HTML characters when parsing HTML email
* GPLA-3576: Ingesting content in-place doesn't handle isSynchronous properly
* GPLA-3457: IsFeedDone doesn't return True for finished feed with no contents
* GPLA-3572: Not handling HTTP 400 error on uploading from URI


# December 9: Support for website mapping, web page screenshots, Groq Llama 3.3 model, bug fixes

### New Features

* :bulb: Graphlit now supports mapping a website with the `mapWeb`mutation. You can provide a URL to a website, and the query will return a list of URLs based on the sitemap.xml (or sitemap-index.xml) file, at or underneath the provided URL.
* :bulb: Graphlit now supports the generation of web page screenshots with the `screenshotPage`mutation. By providing the URL of a web page, and optionally, the maximum desired height of the screenshot, we will screenshot the webpage and ingest it automatically as content.  You can provide an optional workflow, which will be applied to the ingested image content, for operations like generating image descriptions with a vision LLM.
* :bulb: Graphlit now supports the direct summarization of text with the `summarizeText`mutation. By providing the desired summarization strategy, we will summarize the text (i.e. bullet points, social media posts) and return the summarization.
* :bulb: Graphlit now supports the direct extraction of text with the `extractText`mutation. By providing the LLM tool definitions and an optional LLM specification, we will prompt the desired LLM (or OpenAI GPT-4o, by default) to invoke the provided tools, and return the JSON responses from the LLM tool calling.
* Graphlit now supports the latest Groq Llama 3.3 model, with the model enum `LLAMA_3_3_70B`.
* We have updated Cohere reranking to use the latest Cohere `rerank-v3.5`model by default.
* :zap: We have added a new `flattenCitations`field to the `ConversationStrategyInput`type.  By assigning this field to True, when calling `promptConversation,`we will combine multiple citations from the same content into a single citation.
* :zap: For Microsoft email, Microsoft Teams and OneDrive feeds, we have added the `clientId`and `clientSecret`fields as required feed properties. These properties must be assigned, in addition to the `refreshToken`field for proper authentication to the Microsoft Graph API used by these feeds. ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2024_12_09_Locate_Microsoft_Emails_by_Organization.ipynb))

### Bugs Fixed

* GPLA-3492: Not finding sitemap at parent web path
* GPLA-3500: Failed to handle mismatch of Deepgram model/language


# December 1: Support for retrieval-only RAG pipeline, bug fixes

### New Features

* :bulb: Graphlit now supports formatting of LLM-ready prompts with our RAG pipeline, via the new `formatConversation` and `completeConversation` mutations.  This is valuable for supporting LLM streaming by directly calling the LLM from your application, and using Graphlit for RAG retrieval and conversation history. ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2024_12_01_OpenAI_LLM_Streaming.ipynb))
* We have added support for inline hyperlinks in extracted text from documents and web pages.

### Bugs Fixed

* GPLA-3466: Owner ID should accept any non-whitespace string
* GPLA-3458: Not getting Person-to-Organization edges from entity extraction


# November 24: Support for direct LLM prompt, multi-turn image analysis, bug fixes

### New Features

* :bulb: Graphlit now supports multi-turn analysis of images with the `reviseImage` and `reviseEncodedImage` mutations.  You can provide an LLM prompt and either a URI or Base-64 encoded image and MIME type, along with an optional LLM specification.  This can be used for analyzing any image and having a multi-turn conversation with the LLM to revise the output from the LLM. ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2024_11_24_Multi_turn_Analysis_of_Image.ipynb))
* :bulb: Graphlit now supports directly prompting an LLM with the `prompt` mutation, bypassing any RAG content retrieval, while providing an optional list of previous conversation messages.  This also accepts an optional LLM specification. ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2024_11_24_Directly_Prompt_LLM_via_Conversation_Messages.ipynb))
* We have added support for the new Mistral Pixtral Large model, with `PIXTRAL_LARGE` model enum, which can be used with LLM completion or entity extraction LLM specifications.
* We have added support for the OpenAI 2024-11-20 version of GPT-4o, with `GPT4O_128K_20241120` model enum.
* :zap: We have added Microsoft Entra ID (fka Azure Active Directory) `clientId` and `clientSecret` properties to the `SharePointFeedPropertiesInput` type, which are now required when creating a SharePoint feed using user authentication with `refreshToken` property. ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2024_11_25_SharePoint_to_RAG.ipynb))

### Bugs Fixed

* GPLA-3438: Not filtering on desktop presentation when scraping web pages
* GPLA-3340: Failed to parse invalid JSON from extracted PDF page
* GPLA-3427: Not formatting extracted tables properly from Sonnet 3.5


# November 16: Support for image description, multi-turn text summarization

### New Features

* :bulb: Graphlit now supports multi-turn summarization of text with the `reviseText` mutation.  You can provide an LLM prompt and text string, along with an optional specification.  This can be used for summarizing any raw text and having a multi-turn conversation with the LLM to revise the output from the LLM.  ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2024_11_11_Multi_turn_Summarization.ipynb))
* :bulb: Graphlit now supports image descriptions using vision LLMs, without needing to ingest the image first.  With the new `describeImage` mutation, which takes a URI, and `describeEncodedImage` mutation, which takes a Base-64 encoded image and MIME type, you can use any vision LLM to prompt an image description.  These mutations accept an optional specification, where you can select your vision LLM.  If not provided, OpenAI GPT-4o will be used. ([Colab Notebook Example](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2024_11_16_Describe_Image_with_Vision_LLM.ipynb))


# November 10: Support for web search, multi-turn content summarization, Deepgram language detection

### New Features

* :bulb: Graphlit now supports web search with the `searchWeb` mutation.  You can select the search service, either Tavily or Exa.AI, and provide the search query and number of search results to be returned.  This is different than the web search feed, in that `searchWeb` returns the relevant text from the web page and the web page URL from each search hit, but does not ingest each of the web pages. This new mutation is optimized to be used from within an LLM tool.
* :bulb: Graphlit now supports multi-turn summarization of content with the `reviseContent` mutation.  You can provide an LLM prompt and a content reference, along with an optional specification.  This can be used for summarizing any content (documents, web pages, audio transcripts, etc.), and having a multi-turn conversation with the LLM to revise the output from the LLM.  Internally, this creates a conversation locked to a single piece of content.  This works especially well with the OpenAI o1-preview and o1-mini models, because they provide a longer LLM output from each turn.
* Graphlit now supports the configuration of the Deepgram transcription `language`, and whether `detectLanguage` is enabled in `DeepgramAudioPreparationPropertiesInput`.  Language detection is now enabled by default, and can be disabled by setting `detectLanguage` to false.
* :zap: We have added a `requireTool` option to `promptConversation` mutation, so you can control whether the LLM must call one of the provided tool, or if tool calling is optional.
* :zap: For accounts created after Nov 8, 2024, we have lowered the credits quota on the Free tier from 1000 credits to 100 credits, and now offer unlimited feeds on the Hobby Tier.
* :zap: The Graphlit Data API will now return HTTP 402 (Payment Required) when you have exceeded the credits quota on the free tier.  You must upgrade to the Hobby Tier (or higher) to continue using the API, once the credits quota has been reached.


# November 4: Support for Anthropic Claude 3.5 Haiku, bug fixes

### New Features

* Graphlit now supports the latest Anthropic Haiku 3.5 model, with the model enum `CLAUDE_3_5_HAIKU_20241022`.
* :zap: Once a project has hit the free tier quota, we will now automatically disable all feeds.  Once the project has been upgraded to a paid tier, you can use the `enableFeed` mutation to re-enable your existing feeds to continue ingestion.
* :zap: We have added the `disableFallback` flag to the `RetrievalStrategyInput` type, so you can disable the default behavior of falling back to the previous conversation's contents, or worst-case, falling back to the most recently uploaded content.  By setting `disableFallback` to true, conversations will only attempt to retrieve contents based on the provided `filter` and/or `augmentedFilter` properties.

### Bugs Fixed

* GPLA-3367: Not extracting text from HTML button element


# October 31: Support for simulated tool calling, bug fixes

### New Features

* Graphlit now supports simulated tool calling for LLMs which don't natively support it, such as OpenAI o1-preview and o1-mini.  Tool schema will be formatted into the LLM prompt context, and tool responses are parsed out of the JSON formatted response.
* :zap: Given customer feedback, we have lowered the vector and hybrid thresholds used by the semantic search.  Previously, some content at a low relevance was being excluded from the semantic search results.  Now, more low-relevance content will be included in the results, used by the RAG pipeline.  Reranking can be used to sort the search results for relevance.

### Bugs Fixed

* GPLA-3357: Not extracting all images from PDF, and should filter out single-color images.


# October 22: Support for latest Anthropic Sonnet 3.5 model, Cohere image embeddings

### New Features

* Graphlit now supports the latest Anthropic Sonnet 3.5 model (released 10/22/2024).  We have added date-versions model enums for the Anthropic models: `CLAUDE_3_5_SONNET_20240620`, `CLAUDE_3_5_SONNET_20241022`, `CLAUDE_3_HAIKU_20240307`, `CLAUDE_3_OPUS_20240229`, `CLAUDE_3_SONNET_20240229`. The existing model enums will target the latest released models, as specified by Anthropic.
* Graphlit now supports image embeddings using the [Cohere Embed 3.0 models](https://docs.cohere.com/changelog/embed-v3-is-multimodal).


# October 21: Support OpenAI, Cohere, Jina, Mistral, Voyage and Google AI embedding models

### New Features

* :bulb: Graphlit now supports the configuration of image and text embedding models, at the Project level.  You can create an embedding specification for a text or image embedding model, and then assign that to the Project, and all further embedding requests will use that embedding model.  See this [Colab notebook](https://colab.research.google.com/github/graphlit/graphlit-samples/blob/main/python/Notebook%20Examples/Graphlit_2024_10_21_Configure_Embedding_Model.ipynb) for an example of how to configure the project.
* :bulb: Graphlit now supports the OpenAI Embedding-3-Small and Embedding-3-Large, Cohere Embed 3.0, Jina Embed 3.0, Mistral Embed, and Voyage 2.0 and 3.0 text embedding models.  Graphlit also now supports Jina CLIP image embeddings, which are used by default for image search.
* Graphlit now supports the `chunkTokenLimit` property in Specifications, which specifies the number of tokens for each embedded text chunk.  If this is not configured, Graphlit uses 600 tokens for each embedded text chunk.
* Graphlit now supports the Voyage reranking model.
* Graphlit now supports the `ingestTextBatch` mutation, which accepts an array of text and name pairs, and will asynchronously ingest these into content objects.
* :zap: We have moved the `chunkTokenLimit` property from the Workflow storage embeddings strategy to the Specification object.  The Workflow `storage` property has now been deprecated.
* :zap: We have deprecated the `openAIImage` property from Workflow entity extraction properties. Use the `modelImage` property instead. 

{% hint style="warning" %}
Once a text embedding model has been updated at the project level, any content, conversations or observed entities will no longer be semantically searchable. 

Text embeddings are not compatible across models, so you will need to delete and reingest any content, or recreate conversations or knowledge graph entities, with the new embedding model to become searchable.
{% endhint %}


# October 9: Support for GitHub repository feeds, bug fixes

### New Features

* :bulb: Graphlit now supports GitHub feeds, by providing the repository owner and name similar to GitHub Issues feeds, and will ingest code files from any GitHub repository.

### Bugs Fixed

* GPLA-3262: Missing row separator in table markdown formatting


# October 7: Support for Anthropic and Gemini tool calling

### New Features

* :bulb: Graphlit now supports tool calling with Anthropic and Google Gemini models.
* :zap: We have removed the `uri` property for tools from `ToolDefinitionInput`, such that inline webhook tools are no longer supported.  Now you can define any external tools to be called, and those can support sync or async data access to fulfill the tool call.


# October 3: Support tool calling, ingestBatch mutation, Gemini Flash 1.5 8b, bug fixes

### New Features

* :bulb: Graphlit now supports the `ingestBatch` mutation, which accepts an array of URIs to files or web pages, and will asynchronously ingest these into content objects.
* :bulb: Graphlit now supports the `continueConversation` mutation, which accepts an array of called tool responses. Also, `promptConversation` now accepts an array of tool definitions. When tools are called by the LLM, the assistant message returned from `promptConversation` will have a list of `toolCalls` which need to responded to from your calling code.  These responses are to be provided back to the LLM via the `continueConversation` mutation.
* :bulb: Graphlit now supports tool calling with OpenAI, Mistral, Deepseek, Groq, and Cerebras model services.  Anthropic, Google Gemini and Cohere support will come later.
* Added support for prefilled user and assistant messages with `createConversation` mutation. Now you can send an array of messages when creating a new conversation, which will bootstrap the conversation with the LLM.  These must be provided in user/assistant pairs.
* Added support for [Google Gemini Flash 1.5 8b](https://developers.googleblog.com/en/gemini-15-flash-8b-is-now-generally-available-for-use/) model.
* :zap: We have deprecated the `tools` property in the Specification object. These will be removed at a later date.  Tools are now to be sent directly to the `extractContents` and `promptConversation` mutations.

### Bugs Fixed

* GPLA-3207: Models shouldn't be required on update specification call
* GPLA-3220: Don't send system prompt with OpenAI o1 models


# September 30: Support for Azure AI Inference models, Mistral Pixtral and latest Google Gemini models

### New Features

* :bulb: Graphlit now supports the [Azure AI Model Inference API](https://learn.microsoft.com/en-us/azure/ai-studio/reference/reference-model-inference-api?tabs=python) (aka Models as a Service) model service which offers serverless hosting to many models such as Meta Llama 3.2, Cohere Command-R, and many more.  For Azure AI, all models are 'custom', and you will need to provide the serverless endpoint, API key and number of tokens accepted in context window, after provisioning the model of your choice.
* We have added support for the multimodal Mistral Pixtral model, under the model enum `PIXTRAL_12B_2409`.
* We have added versioned model enums for Google Gemini, so you can access `GEMINI_1_5_FLASH_001`, `GEMINI_1_5_FLASH_002`, `GEMINI_1_5_PRO_001` and `GEMINI_1_5_PRO_002`.


# September 26: Support for Google AI and Cerebras models, and latest Groq models

### New Features

* :bulb: Graphlit now supports the [Cerebras](https://cerebras.ai/) model service which offers the `LLAMA_3_1_70B` and `LLAMA_3_1_8B` models.
* :bulb: Graphlit now supports the [Google AI](https://ai.google.dev/) model service which offers the `GEMINI_1_5_PRO` and `GEMINI_1_5_FLASH` models.
* We have added support for the latest Groq Llama 3.2 preview models, including `LLAMA_3_2_1B_PREVIEW`, `LLAMA_3_2_3B_PREVIEW`, `LLAMA_3_2_11B_TEXT_PREVIEW`, and `LLAMA_3_2_90B_TEXT_PREVIEW`.  We have also added support for the Llama 3.2 multimodal model `LLAMA_3_2_11B_VISION_PREVIEW.`
* We have added a new `specification` parameter to the `promptConversation` mutation. Now you can specify your initial specification for a new conversation, or update an existing conversation, without requiring additional API calls.
* :zap: We have changed the retrieval behavior of the `promptConversation` mutation. Now, if no relevant content was found via vector-based semantic search (given the user prompt), we will fallback to any relevant content from the message in the conversation. If there was no content from the conversation to fallback to, we will fallback to the last ingested content in the project. This solves an issue where a first prompt like 'Summarize this' would find no relevant content.  Now it will fallback to retrieve the last ingested content.
* :zap: We have renamed the Groq model enum from `LLAVA_1_5_7B` to `LLAVA_1_5_7B_PREVIEW.`

### Bugs Fixed

* GPLA-3083: Not sending custom instructions/guidance with extraction prompt
* GPLA-3146: Filtering Persons by email not working
* GPLA-3171: Not failing on deprecated OpenAI model
* GPLA-3158: Summarization not using revision strategy


# September 3: Support for web search feeds, model deprecations

### New Features

* :bulb: Graphlit now supports web search feeds, using the [Tavily](https://tavily.com/) and [Exa.AI](https://exa.ai/) web search APIs. You can choose the `SEARCH` feed type, and assign your search `text` property, and we will ingest the referenced web pages from the search results.  Optionally, you can select the search service via the `serviceType` property under `search` feed properties.  By default, Graphlit will use the Tavily API.
* :zap: We have deprecated these OpenAI models, according to the future support OpenAI is providing to these legacy models: `GPT35_TURBO`, `GPT35_TURBO_0613`, `GPT35_TURBO_16K`, `GPT35_TURBO_16K_0125`, `GPT35_TURBO_16K_0613`, `GPT35_TURBO_16K_1106`, `GPT4`, `GPT4_0613`, `GPT4_32K`, `GPT4_32K_0613`, `GPT4_TURBO_VISION_128K`, and `GPT4_TURBO_VISION_128K_1106`.  We suggest using GPT-4o or GPT-4o Mini instead.

### Bugs Fixed

* GPLA-2523: Can't ingest from same feed URI multiple times and wait on isFeedDone


# September 1: Support for FHIR enrichment, latest Cohere models, bug fixes

### New Features

* :bulb: Graphlit now supports entity enrichment from [Fast Healthcare Interoperability Resources (FHIR)](https://en.wikipedia.org/wiki/Fast_Healthcare_Interoperability_Resources) servers. You can provide the `endpoint` for a FHIR server, and Graphlit will enrich medical-related entities from the data found in the FHIR server.
* Added support for latest Cohere models (`COMMAND_R_202408`, `COMMAND_R_PLUS_202408)` and added datestamped model enums for the previous versions (`COMMAND_R_202403`, `COMMAND_R_PLUS_202404`).  The latest model enums (`COMMAND_R` and `COMMAND_R_PLUS`) currently point to the models (`COMMAND_R_202403` and `COMMAND_R_PLUS_202404`) as specified by the Cohere API.
* Added support for the latest Azure AI Document Intelligence v4.0 preview API (2024-07-31), now used by default.
* :zap: We have changed the name of the `LinkReferenceType` to `LinkReference` to follow the existing data model standard.

### Bugs Fixed

* GPLA-3120: LLM is adding source tags to end of completed messages
* GPLA-3133: Failed to load sitemap on child page of website.


# August 20: Support for medical entities, Anthropic prompt caching, bug fixes

### New Features

* :bulb: Graphlit now supports the extraction of medical-related entities: MedicalStudy, MedicalCondition, MedicalGuideline, MedicalDrug, MedicalDrugClass, MedicalIndication, MedicalContraindication, MedicalTest, MedicalDevice, MedicalTherapy, and MedicalProcedure.
* :bulb: Graphlit now supports medical-related entities in GraphRAG, and via API for queries and mutations.
* Added support for [Anthropic prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching#how-prompt-caching-works). When using Anthropic Sonnet 3.5 or Haiku 3, Anthropic will now cache the entity extraction and LLM document preparation system prompts, which saves on token cost and increases performance.

### Bugs Fixed

* GPLA-3104: Should default search type to VECTOR, when performing entity similarity filter.
* GPLA-3112: Empty PDF fails entity extraction.


# August 11: Support for Azure AI Document Intelligence by default, language-aware summaries

### New Features

* Added support for language-aware summaries when using LLM-based document extraction.  Now the summaries for tables and sections generated by the LLM will follow the language of the source text.
* Added support for language-aware entity descriptions with using LLM-based entity extraction. Now the entity descriptions generated by the LLM will follow the language of the source text.
* :zap: We have changed the default document preparation method to use Azure AI Document Intelligence, rather than our built-in document parsers.  We have found that the fidelity of Azure AI is considerably better for complex PDFs, and provides better support for table extraction, so we have made this the default. **Note:** this does come with increased credit usage per-page, for PDF, DOCX and PPTX documents, but the quality of the extracted documents are noticeably higher for use in RAG pipelines.

### Bugs Fixed

* GPLA-3070: Not getting slide count assigned to metadata for PPTX files.


# August 8: Support for LLM-based document extraction, .NET SDK, bug fixes

### New Features

* :bulb: Graphlit now supports LLM-based document preparation, using vision-capable models such as OpenAI GPT-4o and Anthropic Sonnet 3.5.  This is available via the `MODEL_DOCUMENT` preparation service type, and you can assign a custom`specification` object and bring your own LLM keys.
* :bulb: Graphlit now provides an open source .NET SDK, supporting .NET 6 and .NET 8 (and above).  SDK package can be found on [Nuget.org](https://www.nuget.org/packages/Graphlit).  Code samples can be found on [GitHub](https://github.com/graphlit/graphlit-samples/tree/main/dotnet).
* Added `identifier` property to `Content` object for mapping content to external database identifiers.  This is supported for content filtering as well.
* Added support for Claude 3 vision models for image-based entity extraction, using the `MODEL_IMAGE` entity extraction service.
* Added context augmentation to conversations, via the `augmentedFilter` property on the `Conversation` object.  Any content which matches this augmented filter will be injected into the LLM prompt content, without needing to be related by vector similarity to the user prompt.  This is useful for specifying domain knowledge which should always be referenced by the RAG pipeline.
* Added support for the latest snapshot of OpenAI GPT-4o, with the model enum `GPT4O_128K_20240806.` 
* Added reranking of related entities, when preparing the LLM prompt context for GraphRAG.  If reranking is enabled, the metadata from the related entities will be reranked with the same reranker assigned to the conversation specification.
* :zap: We have changed the type of the `duration` field in the `AudioMetadata` and `VideoMetadata` types to be `TimeSpan` rather than `string`, as to be more consistent with the rest of the API data model.

### Bugs Fixed

* GPLA-2884: Support retry on HTTP 529 (Overloaded) error from Anthropic API.


# July 28: Support for indexing workflow stage, Azure AI language detection, bug fixes

### New Features

* Added `indexing` workflow stage. This provides for configuration of indexing services, which may infer metadata from the content.
* Added `AZURE_AI_LANGUAGE` content indexing service, which supports inferring the language of extracted text or transcript.
* Added support for `language` content metadata.  This returns a list of languages in ISO 639-1 format, which may have been inferred from the extracted text or transcript.
* Added support for `MODEL_IMAGE` extraction service.  This provides integration with vision models beyond those provided by OpenAI.  You can assign a custom specification and bring-your-own API key for image extraction models.
* :zap: We have deprecated the `OPENAI_IMAGE` service type, and developers should now use the LLM image service instead.
* :zap: We have removed the `language` field from `AudioMetadata` type, which has been replaced by the new `LanguageMetadata` type.

### Bugs Fixed

* GPLA-2987: Extracting text with Azure Doc Intelligence does not extract hyperlinks


# July 25: Support for Mistral Large 2 & Nemo, Groq Llama 3.1 models, bug fixes

### New Features

* :bulb: Graphlit now supports the Mistral Large 2 and Mistral Nemo models. The existing `MISTRAL_LARGE` model enum now will use Mistral Large 2.
* :bulb: Graphlit now supports the Llama 3.1 8b, 70b and 405b models on Groq.  (Note, these are rate-limited according to Groq's platform constraints.)
* Added support for revision strategy on data extraction specifications.  Now you can prompt the LLM to revise its previous data extraction response, similar to the existing completion revision strategy.
* Added `version` property for `AzureDocumentPreparationProperties` type for assigning the [API version](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/overview?view=doc-intel-4.0.0) used by Azure AI Document Intelligence.   By default, Graphlit will continue to use the v4.0 (Preview) API version, but you can override this to assign `version` to `V2023_10_31` to use the v3.1 (GA) API version instead.  For some documents, the General Availability (GA) version of the API can provide better results.

### Bugs Fixed

* GPLA-2988: Not extracting hyperlinks from Office documents.


# July 19: Support for OpenAI GPT-4o Mini, BYO-key for Azure AI, similarity by summary, bug fixes

### New Features

* :bulb: Graphlit now supports the [OpenAI GPT-4o Mini](https://platform.openai.com/docs/models/gpt-4o-mini) model, with 16k output tokens.
* :bulb: Graphlit now supports 'bring-your-own-key' for Azure AI Document Intelligence models.  We have added a custom `endpoint` and `key` property, which can be assigned to use your own Azure AI resource.
* Updated to use [Jina reranker v2](https://jina.ai/reranker/) (jina-reranker-v2-base-multilingual) by default.
* Updated to assign the `summary`, `bullets`, etc properties when calling `summarizeContents` mutation.  Now when summarizing contents, we will store the resulting summary in the content itself, in addition to returning the summarized results.
* Added `relevance` property to all entity types, which will be assigned when searching for these entities.  Entity results will be sorted (descending) by this search relevance score. 
* Added the ability to manually update `summary`, `bullets`, etc. properties when calling the `updateContent` mutation.
* Added `offset` property to AtlassianJiraFeedProperties, so the timezone offset can be properly assigned for paging of the Jira feed.  (Defaults to zero offset, i.e. UTC.)  Jira does not store dates in UTC format, and the timestamps are based on the server timezone of the hosted Jira instance.  By assigning the timezone offset with the Jira feed, we can reliably page the updated date timestamps from the Jira API.
* :zap: We have changed the content similarity search behavior to find similar content by summary, rather than text of the document, when a summary has been previously generated.  For long documents, this will provide a more accurate similarity, rather than comparing on the first few pages of text in a document.
* :zap: We have changed the behavior of assigning `offset` in the entity filter objects for paging through entities.  If using vector or hybrid search, this offset will be ignored (i.e. zero offset).  Paging will not be supported through vector or hybrid search results. For keyword search, the offset will continue to be used, along with the `limit` property, to provide paging through the search results.  We have made this change because we have found that index-based paging is not reliable with our vector/hybrid search approach.  We are investigating ways to support this reliably with vector/hybrid search in the future.

### Bugs Fixed

* GPLA-2915: Add retry on OpenAI API HTTP 524 error (gateway timeout).
* GPLA-2908: Not paging through Jira feed correctly.
* GPLA-2917: Search by similar content is not giving expected results on long documents.
* GPLA-2244: Keyword search not finding text in latter part of long PDF.


# July 4: Support for webhook Alerts, keywords summarization, Deepseek 128k context window, bug fixes

### New Features

* :bulb: Graphlit now supports webhook Alerts.  In addition to Slack notifications, you can now receive an HTTP POST webhook with the results of the published text (or text and audio URI) from a prompted alert.
* Updated the Deepseek chat and coder models to support a [128k token context window](https://platform.deepseek.com/api-docs/#models).
* Added `customSummary` property to `Content` object, which returns the custom summary generated via preparation workflow.
* Added `keywords` summarization type, which is now stored in `keywords` property in `Content` object.
* Added `slackChannels` query, which returns the list of Slack channels from the workspace authenticated by the Slack bot token.
* :zap: We have changed the response from the `credits` query to return a single `ProjectCredits` object, rather than the list of correlated objects previously returned.  The `credits` response now covers all credit usage over the time period specified.

### Bugs Fixed

* GPLA-2874: Processing entities is taking longer than 30min for 300+ page PDF
* GPLA-2875: Messages in queue expiring too early
* GPLA-2881: Feed read count increasing, after hitting read limit
* GPLA-2884: Need to handle Anthropic 'overloaded' API response
* GPLA-2906: JIRA issue identifier not assigned to issue metadata


# June 21: Support for the Claude 3.5 Sonnet model, knowledge graph semantic search, and bug fixes

### New Features

* :bulb: Graphlit now supports the Anthropic Claude 3.5 Sonnet model, which can be assigned with the `CLAUDE_3_5_SONNET` model enum.
* :bulb: Graphlit now supports semantic search of observable entities in the knowledge graph, such as Person, Organization and Place.  These entity types will now have vector embeddings created from their enriched metadata, and support searching by similar text, and searching by similar entities.
* :zap: We have changed the Google Drive and Google Email feed properties to require the Google OAuth client ID and client secret, along with the existing refresh token, for proper authentication against Google APIs.
* :zap: We have added a credits quota on the Free Tier.  Once 1000 credits have been used on the Free Tier, no more content can be ingested, and an upgrade to a paid tier is required.  Customers will receive an email when the credits, storage or contents quota has been reached.

### Bugs Fixed

* GPLA-2837: Failed to ingest LinkedIn page as Web feed
* GPLA-2831: Zero-byte file was left in Indexed state
* GPLA-2834: Not reading any files from Azure blob feed with space in prefix
* GPLA-2828: Better handling for files with unknown (or missing) file extensions


# June 9: Support for Deepseek models, JSON-LD webpage parsing, performance improvements and bug fixes

### New Features

* :bulb: Graphlit now supports [Deepseek](https://platform.deepseek.com/api-docs/#models) LLMs for prompt completion.  We offer the `deepseek-chat` and `deepseek-coder` models.
* :bulb: Graphlit now supports parsing embedded JSON-LD from web pages.  If a web page contains 'script' tags with JSON-LD, we will automatically parse and inject into the knowledge graph.
* :zap: We have changed the default model for entity extraction and image completions to be OpenAI GPT-4o.  This provides faster performance and better quality output.
* :zap: We have changed the behavior of knowledge graph generation, from a prompted conversation, to be opt-in.  In order to receive the graph's nodes and edges with the response, you will now need to set `generateGraph` to True in the specification's `graphStrategy` object.  This provides improved performance when the graph is not needed for visualization.
* Added `thing` property for observable entities, which returns the JSON-LD metadata associated with the entity.
* Added regex-based filtering for URI paths during feed ingestion, link crawling, and workflow filtering.  You can assign regex patterns in `allowedPaths` and `excludedPaths`.
* Added `observableLimit` to configure the limit of how many observed entities will be added to the GraphRAG context, defaults to 1000.
* Added `prompt` to `suggestConversation` mutation, which allows customization of the followup question generation.
* Updated `suggestConversation` to incorporate the past conversation message history, in addition to the filtered set of content sources.
* :fire:  We have improved performance in knowledge graph retrieval and generation, via better parallelization and batching.

### Bugs Fixed

* GPLA-2748: Optimize the retrieval performance of observed entities during GraphRAG
* GPLA-2732: Invalid user-provided URI causing parsing exception
* GPLA-2666: Shouldn't require tenant ID for Microsoft email or Teams
* GPLA-2772: Not returning labels or categories from graph in API
* GPLA-2762: Failed to extract spreadsheet images
* GPLA-2687: Email to/from not getting added as observations on emails
* GPLA-2738: API is returning 'audio' metadata from podcast HTML document


# May 15: Support for GraphRAG, OpenAI GPT-4o model, performance improvements and bug fixes

### New Features

* :bulb: Graphlit now supports GraphRAG, where the extracted entities in the knowledge graph can be added as additional context to your RAG con,versation.  Also, with GraphRAG, entities can be extracted from the user prompt, and used as additional content filters - or can be used to query related content sources, which are combined with the vector search results.  This can be configured by specifying your `graphStrategy` in the Specification object.
* :bulb: Graphlit now supports LLM revisions within RAG conversations, where the LLM can be prompted to revise its initial completion response. From our testing, this has been shown to provide 35% more output tokens with higher quality responses.  This can be configured by specifying your `revisionStrategy`, and you can use our built-in revision prompt, or provide a custom one, and specify how many revisions you want the LLM to make.
* :bulb: Graphlit now supports the new OpenAI [GPT-4o ](https://platform.openai.com/docs/models/gpt-4o)model for RAG conversations.
* :zap: We have changed the default model for Conversations to be OpenAI GPT-4o, from Azure OpenAI GPT-3.5 16k.  This provides faster performance and better quality output.
* Added `graph` to `promptConversation` response, so you can visualize or leverage the nodes and edges of the knowledge graph, resulting from the content retrieval.  For example, if a Person and Organization were observed in the cited content sources used by the RAG pipeline, you will get back those entities and their relationship (such as Person 'works-for' Organization).
* Expanded the enriched data from WIkipedia to include the long description of an entity.
* Added `getSharePointLibraries`, `getSharePointFolders`, and `getOneDriveFolders` queries to the API, which can be used to enumerate the storage services.  This makes locating the SharePoint `libraryId` easier, for example.
* Added `getTeams` and `getTeamsChannels` queries to the API for enumerating Microsoft Teams workspaces.
* Added `extractedCount` to the entity extraction connector to limit the number of extracted entities, per entity type.  I.e. if extracted count is 10, it will extract at most ten each of Persons, Organizations, etc.
* :fire:  We have improved performance in several areas: creation of observations after entity extraction, access to cloud storage, rendering the RAG context.
* :fire:  We have optimized the LLM entity extraction process to identify more properties, as well as entity-to-entity relationships.

### Bugs Fixed

* GPLA-2652: Not extracting text from HTML in RSS post
* GPLA-2627: Limit filter only returning half the results
* GPLA-2613: Not properly extracting structured text from JSON/XML formats


# May 5: Support for Jina and Pongo rerankers, Microsoft Teams feed, new YouTube downloader, bug fixes

### New Features

* :bulb: Graphlit now supports the [Jina reranker](https://jina.ai/reranker/) and [Pongo semantic filtering](https://www.joinpongo.com/) (reranking), in the Specification object.  Now you can choose between `COHERE`, `PONGO` and `JINA` for your reranking `serviceType`.
* :bulb: Graphlit now supports Microsoft Teams feeds for reading messages from Teams channels.
* Given changes in YouTube video player HTML, we have rewritten the YouTube downloader to support the new page format.
* Added better handling of HTTP errors when validating URIs.  Previously some websites were returning HTTP 403 (Forbidden) errors when validating their URI, or downloading content.  Now Graphlit is able to scrape these sites, which previously returned errors.
* Added support for updating content metadata in `updateContent` mutation.  Now the video, audio, document, etc. metadata can be updated after the content workflow has finished.
* Added `query_contents_graph` (and `queryContentsGraph`) functions to SDKs, which can be used to return nodes and edges from knowledge graph for visualization.
* :zap: Citation indices have been changed to be one-based from zero-based.  For example, you will now see "This is a citation. \[1]" as the first citation in the list.
* :zap: Added `isSynchronous` flag to deleteAll and multiple delete mutations.  By default, bulk delete operations are now asynchronous (and completed after the mutation returns), unless the isSynchronous flag is set to true.
* :zap: Added missing count mutations, such as `countAlerts`, `countFeeds`, etc.
* :zap: Renamed `query_content_facets` to `query_contents_facets` in Python SDK
* :zap: Renamed `queryContentFacets` to `queryContentsFacets` in Node.js SDK

### Bugs Fixed

* GPLA-2544: Page `relevance` not filled-in in all situations
* GPLA-2546: Not extracting links from PDF with Azure AI Doc Intelligence
* GPLA-2557: Sporadically returning HTTP 500 from GraphQL API
* GPLA-2573: Failed to re-ingest content which was deleted immediately after initial ingestion
* GPLA-2575: Not validating for empty (non-null) parameters in mutations
* GPLA-2578: Need to handle invalid JSON from LLMs; improper escaping or formatting
* GPLA-2585: Failed to ingest encoded file with colon (:) in name


# April 23: Support for Python and TypeScript SDKs, latest OpenAI, Cohere & Groq models, bug fixes

### New Features

* :bulb: Graphlit now supports a native Python SDK, using Pydantic types. The Python SDK is code-generated from the current GraphQL schema, but does not require GraphQL knowledge. You can find the latest PyPi package [here](https://pypi.org/project/graphlit-client/).  The Streamlit [sample applications](https://github.com/graphlit/graphlit-samples/tree/main/python) have been updated to use the new Python SDK.
* :bulb: Graphlit now supports a native Node.js SDK, using TypeScript types. The Node.js SDK is code-generated from the current GraphQL schema, but does not require GraphQL knowledge. You can find the latest NPM package [here](https://www.npmjs.com/package/graphlit-client).
* :bulb: Graphlit now supports the 2024-04-09 models in the OpenAI model service. `GPT4_TURBO-128K` will give the latest OpenAI GPT-4 model, following this model [list](https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4).  We have added the `GPT4_TURBO_128K_2024_04_09` enum to specify the new model.
* :bulb: Graphlit now supports [LLaMA3 70b, LLaMA3 8b and Gemma 7b models](https://console.groq.com/docs/models) in the Groq model service.
* :bulb: Graphlit now supports the [Command R and Command-R+](https://cohere.com/command) models in the Cohere model service.
* Added support for [Jina reranking](https://jina.ai/reranker/), using the `JINA` reranking model service type in the reranking retrieval strategy.
* Updated the Cohere reranking model to use the latest [v3.0](https://docs.cohere.com/reference/rerank) model.
* Increased the reliability of parsing LLM responses, in cases where they don't follow the JSON schema.
* :zap: Cleaned up nullability of GraphQL parameters, so parameters better reflect if they are required or optional, or allow nulls.
* :zap: Added missing `deleteWorkflows` and `deleteAllCollections` mutations.
* :zap: Split out reranking model service type as `RetrievalModelServiceTypes` enum. 

### Bugs Fixed

* GPLA-2114: Adding content to collections not syncing search index
* GPLA-2511: Failing to render any conversation sources with section retrieval and text content


# April 7: Support for Discord feeds, Cohere reranking, section-aware chunking and retrieval

### New Features

* :bulb: Graphlit now supports Discord feeds.  By connecting to a Discord channel and providing a bot token, you can ingest all Discord messages and file attachments.
* :bulb:  Graphlit now supports [Cohere reranking](https://cohere.com/rerank) after content retrieval in RAG pipeline.  You can optionally use the Cohere rerank model to semantically rerank the semantic search results, before providing as context to the LLM.
* Added support for section-aware text chunking and retrieval.  Now, when using section-aware document preparation, such as Azure AI Document Intelligence, Graphlit will store the extracted text according to the semantic chunks (i.e. sections).  The text for each section will be individually chunked and embedded into the vector index.
* Added support for `retrievalStrategy` in Specification type. Graphlit now supports `CHUNK`, `SECTION` and `CONTENT` retrieval strategies.  Chunk retrieval will use the search hit chunk, section retrieval will expand the search hit chunk to the containing section (or page, if not using section-aware preparation).  Content retrieval will expand the search hit chunk to the text of the entire document.
* Added support for `rerankingStrategy` in Specification type. You can now configure the reranking of content sources, using the Cohere reranking model, by assigning `serviceType` to `COHERE`.  More reranking models are planned for the future.
* Added `isSynchronous` flag to content ingestion mutations, such as `ingestUri`, so the mutation will wait for the content to complete the ingestion workflow (or error) before returning.  This is useful for utilizing the API in a Jupyter notebook or Streamlit application, in a synchronous manner without polling. 
* Added `includeAttachments` flag to SlackFeedProperties.  When enabled, Graphlit will automatically ingest any attachments within Slack messages.
* :zap: Added `ingestUri` mutation to replace the now deprecated `ingestPage` and `ingestFile` mutations.  We had seen confusion on when to use one vs the other, and now for any URI, whether it is a web page or hosted PDF, you can pass it to `ingestUri`, and we will infer the correct content ingestion workflow.
* :zap: Removed `includeSummaries` from the ConversationStrategyInput type.  This will re-added in the future as part of the retrieval strategy.
* :zap: Deprecated `enableExpandedRetrieval` in ConversationStrategyInput type.  This is now handled by setting `strategyType` to `SECTION` or `CONTENT` in the RetrievalStrategyInput type.
* :zap: Moved `contentLimit` from ConversationStrategyInput type to RetrievalStrategyInput type. You can optionally assign the `contentLimit` to `retrievalStrategy` which limits the number of content sources leveraged in the LLM prompt context. (Default is 100.)

### Bugs Fixed

* GPLA-2469: Failed to ingest PDF hosted on GitHub
* GPLA-2390: Claude 3 Haiku not adhering to JSON schema
* GPLA-2474: Prompt rewriting should ignore formatting instructions in prompt
* GPLA-2462: Missing line break after table rows
* GPLA-2417: Not extracting images from PPTX correctly 


# March 23: Support for Linear, GitHub Issues and Jira issue feeds, ingest files via Web feed sitemap

### New Features

* :bulb: Graphlit now supports [Linear](https://docs.graphlit.dev/graphlit-data-api/api-reference/feeds/issue-feeds/create-linear-feed), [GitHub Issues](https://docs.graphlit.dev/graphlit-data-api/api-reference/feeds/issue-feeds/create-github-issues-feed) and [Atlassian Jira](https://docs.graphlit.dev/graphlit-data-api/api-reference/feeds/issue-feeds/create-jira-feed) feeds.  Graphlit will ingest issues (aka tasks, stories) from these issue-tracking services as individual content items, which will be made searchable and conversational.
* :bulb: Added support for `ISSUE`content type, which includes metadata such as title, authors, commenters, status, type, project and team.
* :bulb: Added support for default feed read limit.  Now, if you don't specify the `readLimit` property on feeds, it will default to reading 100 content items.  You can override this default by assigning a custom read limit, which has no upper bounds.  However, one-shot feeds much complete within 15 minutes, or they will be stopped automatically.
* Added support for ingesting files referenced in a Web sitemap.  Previously any files (i.e. PDF, MP3) referenced in a sitemap.xml would be ignored.  Now you can optionally enable `includeFiles` in the `WebFeedPropertiesInput` object to have Graphlit ingest non-HTML pages as part of the Web feed.

### Bugs Fixed

* GPLA-2374: Failed to ingest MP4 with large XMP metadata.


# March 13: Support for Claude 3 Haiku model, direct ingestion of Base64 encoded files

### New Features

* :bulb: Graphlit now supports the Claude 3 Haiku model.
* Added support for direct ingestion of Base64 encoded files with the `ingestEncodedFile` mutation.  You can pass a Base64 encoded string and MIME type of the file, and it will be ingested into the Graphlit Platform.
* Added `modelService` and `model` properties to `ConversationMessage` type, which return the model service and model which was used for the LLM completion.


# March 10: Support for Claude 3, Mistral and Groq models, usage/credits telemetry, bug fixes

### New Features

* :bulb: Graphlit now supports a Command-Line Interface (CLI) for directly accessing the Graphlit Data API without writing code.  See the documentation [here](https://docs.graphlit.dev/cli-reference/graphlit-cli).
* :bulb: Graphlit now supports the Groq Platform, and models such as [Mixtral 8x7b](https://console.groq.com/docs/models#mixtral8x7b).
* :bulb: Graphlit now supports Claude 3 Opus and Sonnet models.
* :bulb: Graphlit now supports [Mistral La Plateforme](https://mistral.ai/technology/), and models such as Mistral Small, Medium, and Large and Mixtral 8x7b.
* :bulb: Graphlit now supports the [latest v4 of Azure Document Intelligence](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/whats-new?view=doc-intel-4.0.0\&tabs=csharp#february-2024), including their new models such as Credit Card, Marriage Certificate, and Mortgage documents.
* Added support for detailed usage and credits telemetry via API, with the `usage`, `credits`, `lookupUsage` and `lookupCredits` queries.
* Added support for correlated telemetry, where an optional `correlationId` can be provided with GraphQL queries and mutations, so credits and usage can be tracked across requests.
* Added support for project webhook, which will be called when credits have been consumed by the project.
* Added support for image extraction during DOCX, XLSX, and PPTX document preparation.
* Added `text` and `markdown` properties to Content object, which provide formatted output of extracted text from any content.
* Added more accurate extraction of tables into mezzanine JSON format, across all content types.
* Added `throughput` property to Conversation messages, which returns the tokens/second throughput of LLM.
* :zap: Deprecated `mezzanineUri` property in Content object, which has been replaced by `textUri` and `audioUri`.

### Bugs Fixed

* GPLA-2281: Not extracting table from PPTX file.
* GPLA-2282: Not extracting Markdown tables.
* GPLA-2247: Not extracting relative HTML links properly.
* GPLA-2241: Failed to post Alert to Slack with Markdown format.


# February 21: Support for OneDrive and Google Drive feeds, extract images from PDFs, bug fixes

### New Features

* :bulb: Graphlit now supports OneDrive and Google Drive feeds.  Files can be ingested from OneDrive or Google Drive, including shared drives where the authenticated user has access.  Both OneDrive and Google Drive support the reading of existing files, and tracking new files added to storage with recurrent feeds.
* :bulb: Graphlit now supports email backup files, such as EML or MSG, which will be assigned the `EMAIL` file type.  During email file preparation, we will automatically extract and ingest any file attachments.
* :bulb: Graphlit now automatically extracts embedded images in PDF files, ingests them as content objects, and links them as children of the parent PDF.
* :bulb: Graphlit now supports recursive Notion feeds.  When the `isRecursive` flag is true in the Notion feed properties, we will crawl child pages and databases, and recursively ingest them in addition to the specified pages and databases.
* Added support for assigning `collections` to content ingested with the `ingestPage`, `ingestFile` or `ingestText` mutations.  This saves a step where the content will automatically be added to the collection(s) without requiring another mutation call.
* Added support for the `CODE` file type for a wide variety of source code formats, i.e. Python .py, Javascript .js.  Code files use optimized text splitting for enhanced search and retrieval.
* Added support for `customGuidance` in Specification object, which can be used for injecting a guidance prompt during the RAG process.  For example, you can instruct the LLM to return a default response string if no content sources are found via semantic search.
* Added `tenants` field to Project object, which returns a list of all tenant IDs which have been used to create an entity in Graphlit.
* Added `email` metadata, separate from `document` metadata.  Now emails will contain indexed metadata such as to, from, or subject.
* :zap: The `contents` field for content objects has been replaced with `children` and `parent` fields.  For example, when a ZIP file is unpacked, the unpacked files will be added as children of the ZIP file, and the ZIP file will be the parent of each of the unpacked files.
* :zap: Removed `enableImageAnalysis` field from image preparation properties in workflow object.  Now is enabled by default.
* :zap: Moved `disableSmartCapture` field to preparation workflow stage from page preparation properties.  This is used to disable the use of headless Chrome browser to capture HTML from web pages.  It is enabled by default, and if disabled, Graphlit will simply download the HTML from the web page rather than rendering on headless Chrome browser.

### Bugs Fixed

* GPLA-2099: Failed to ingest ArXiV PDF.  Fixed PDF parsing error.
* GPLA-2174: LLM response is incorrect with conversation history, but no content sources.
* GPLA-2199: ZIP package left in Indexed state after content workflow.


# February 2: Support for Semantic Alerts, OpenAI 0125 models, performance enhancements, bug fixes

### New Features

* :bulb: Graphlit now supports [Semantic Alerts](https://docs.graphlit.dev/api-reference/graphql-data-model/alerts), which allows for LLM summarization and publishing of content, on a periodic basis.  This is useful for generating daily reports from email, Slack or other time-based feeds.  Alerts support the same publishing options, i.e. audio and text, as the `publishContents` mutation.
* :bulb: Graphlit now supports the latest OpenAI 0125 model versions, for GPT-4 and GPT-3.5 Turbo.  We will add support for Azure OpenAI when Microsoft releases support for these.
* Slack feeds now support a listing `type` field, where you can specify if you want `PAST` or `NEW` Slack messages in the feed.
* 🔥 This release provides many performance enhancements, which will speed up the content workflows for ingested content.

### Bugs Fixed

* GPLA-2114: Collections not being added to text embedding index documents.
* GPLA-2063: Not handling hallucinated citations.
* GPLA-1916: Collections not inherited from project-scope into tenant-scope.
* GPLA-2105: Should error on add/remove of contents to/from collections if content does not exist.


# January 22: Support for Google and Microsoft email feeds, reingest content in-place, bug fixes

### New Features

* :bulb: Graphlit now supports Google and Microsoft email feeds.  Email feeds can be created to ingest past emails, or poll for new emails.  Emails create an `EMAIL` content type. Attachment files can optionally be extracted from emails, and will be linked to their parent email content. If assigning a workflow to the feed, the workflow will be applied both to the email content and the extracted attachment files.
* :bulb: Graphlit now supports reingesting content in-place.  The `ingestText`, `ingestPage` and `ingestFile` mutations now take an optional `id` parameter for an existing content object.  If this id is provided, the existing content will be updated from the provided text or URI source, and will restart the assigned workflow.
* Added `restartAllContents` mutation to restart workflow on all partially-ingested contents in project.
* Added `text` field to ConversationCitation type, which returns the relevant text from the content source with the citation.

### Bugs Fixed

* GPLA-1313: Not extracting links from HTML
* GPLA-2030: No text extracted from shapes in PPTX files


# January 18: Support for content publishing, LLM tools, CLIP image embeddings, bug fixes

### New Features

* :bulb: Graphlit now supports content publishing, where documents, audio transcripts and even image descriptions, can be summarized, and repurposed into blog posts, emails or AI-generated podcasts.  With the new `publishContents` mutation, you can configure LLM prompts for summarization and publishing, and assign `specifications` to use different models and/or system prompts for each step in the process.  The published content will be reingested into Graphlit, and can be searched or used for conversations, like any other form of content.
* :bulb: Graphlit now supports publishing conversations as content with the new `publishConversation` mutation.  You can generate text or audio transcripts of your conversations, to be reused in other tools.
* :bulb: Graphlit now supports bulk summarization of contents with the `summarizeContents` mutation.  You can filter a set of content, by feed, by observable or by similar text, and run a set of summarizations across each content in parallel.
* :bulb: Graphlit now supports LLM entity extraction, with the new `MODEL_TEXT` entity extraction service type.  Similar to using Azure Cognitive Service Text Analytics, you can use any OpenAI or Anthropic model for extracting entities from text.  Internally the LLM returns JSON-LD entities, which we convert into Person, Organization, Place, etc. entities and assign `observations` to the extracted content.
* :bulb: Graphlit now supports LLM tools (aka function calls) with OpenAI models.  You can define the tools to be used with the LLM in the `specification` object.  With the new `extractContents` mutation, you can execute a prompt against content using a specification with tools defined.  The mutation will return the JSON arguments assigned by the LLM.
* :bulb: Graphlit now supports callback webhooks for LLM tools.  If you assign a URI in the `ToolDefinition` object, Graphlit will call your webhook the tool name and JSON arguments.  When you respond to the webhook with JSON, we will add that response to the LLM messages, and ask the LLM to complete the original prompt.
* :bulb: Graphlit now supports the selection of the [Deepgram model (such as Meeting, Phonecall or Finance)](https://developers.deepgram.com/docs/model) with the preparation workflow.  Also, you can assign your own Deepgram API key, which will be used for audio transcription using that workflow.
* Added support for CLIP image embeddings using [Roboflow](https://www.roboflow.com), which can be used for similar image search.  If you search for contents by similar contents, we will now use the content's text and/or image embeddings to find similar content.
* Added support for dynamic web page ingestion.  Graphlit now navigates to and automatically scrolls web pages using [Browserless.io](https://www.browserless.io), so we capture the fully rendered HTML before extracting text.  Also, we now support web page screenshots, if enabled with `enableImageAnalysis` property in preparation workflow.  These screenshots can be analyzed with multimodal modals, such as GPT-4 Vision, or can be used to create image embeddings for similar image search.
* Added table parsing when preparing documents.  We now store structured (tab-delimited) text in the JSON text mezzanine which is extracted from documents in the preparation workflow.
* Added reverse geocoding of lat/long locations found in image or other content metadata.  We now store the real-world address with the content metadata, for use in conversations.
* Added assistant messages to the conversation message history provided to the LLM.  Originally we had included only user messages, but now we are formatting both user and assistant messages into the LLM prompt for conversations.
* Added new chunking algorithm for text embeddings.  We support semantic chunking at the page or transcript segment level, and now will create embeddings from smaller sized text chunks per page or segment.
* Added content metadata to text and image embeddings.  To provide better context for the text embeddings, we now include formatted content metadata, which includes fields like title, subject, author, or description.  For emails, we include to, from, cc, and bcc fields.
* Added helper mutations `isContentDone` and `isFeedDone` which can be used for polling completion of ingested content, or all content ingested by a feed.
* Added richer image descriptions generated by the GPT-4 Vision model.  Now these provide more useful detail.
* Added validation of extracted hyperlinks.  Now we test the URIs and remove any inaccessible links during content enrichment.
* Added `deleteContents`,  `deleteFeeds`,  and `deleteConversations` mutations for multi-deletion of contents, feeds or conversations.
* Added `deleteAllContents`,  `deleteAllFeeds`,  and `deleteAllConversations` mutations for bulk, filtered deletion of entities.  You can delete all your contents, feeds, or conversations in your project, or a filtered subset of those entities.
* :information\_source: Starter tier now has a higher content limit of 100K content items.
* :zap: In the `OpenAIImageExtractionProperties` type, the `detailMode` field was renamed to `detailLevel`.
* :zap: Each `SummarizationStrategy` object now accepts the `specification` which is used by the summarization, rather than being assigned at the preparation workflow stage.
* :zap: `addCollectionContents` and `removeCollectionContents` mutations have been deprecated in favor of `addContentsToCollections` and `removeContentsFromCollection` mutations.

### Bugs Fixed

* GPLA-1846: Parse Markdown headings into mezzanine JSON
* GPLA-1779: Not returning SAS token with mezzanine, master URIs
* GPLA-1348: Summarize text content, not just file content
* GPLA-1297: Not assigning content error message on preparation workflow failure


# December 10: Support for OpenAI GPT-4 Turbo, Llama 2 and Mistral models; query by example, bug fixes

### New Features

* :bulb: Graphlit now supports the [OpenAI GPT-4 Turbo 128k](https://openai.com/research/gpt-4v-system-card) model, both in Azure OpenAI and native OpenAI services.  Added new model enum `GPT4_TURBO_VISION_128K`.
* :bulb: Graphlit now supports Llama 2 7b, 13b, 70b models and Mistral 7b model, via [Replicate](https://replicate.com/).  Developers can use their own Replicate API key, or be charged as credits for Graphlit usage.
* :bulb: Graphlit now supports the [Anthropic Claude 2.1](https://www.anthropic.com/index/claude-2-1) model. Added new model enum `CLAUDE_2_1`.
* :bulb: Graphlit now supports the [OpenAI GPT-4 Vision](https://openai.com/research/gpt-4v-system-card) model for image descriptions and text extraction.  Added new model enum `GPT4_TURBO_VISION_128K`. See usage example in ["Multimodal RAG" blog post](https://www.graphlit.com/blog/multimodal-rag-insurance-insights).
* Added query by example to `contents` query.  Developers can specify one or more example contents, and query will use vector embeddings to return similar contents.
* Added query by example to `conversations` query.  Developers can specify one or more example conversations, and query will use vector embeddings to return similar conversations.
* Added vector search support for `conversations` queries.  Developers can provide search text which will use vector embeddings to return similar conversations.
* Added `promptSpecifications` mutation for directly prompting multiple models.  This can be used to evaluate prompts against multiple models or compare different specification parameters in parallel.
* Added `promptStrategy` field to Specification, which supports multiple strategy types for preprocessing the prompt before being sent to the LLM model.  For example, `REWRITE` prompt strategy will ask LLM to rewrite the incoming user prompt based on the previous conversation messages.
* Added `suggestConversation` mutation, which returns a list of suggested followup questions based on the specified conversation and related contents.  This can be used to auto-suggest questions for chatbot users.
* Added new summarization types: `CHAPTERS`, `QUESTIONS` and `POSTS`.   See usage examples in the ["LLMs for Podcasters" blog post](https://www.graphlit.com/blog/llms-for-podcasters).
* Added versioned model enums such as `GPT4_0613` and `GPT35_TURBO_16K_1106`.  Without version specified, such as `GPT35_TURBO_16K`, Graphlit will use the latest production model version, as defined by the LLM vendor.
* Added `lookupContents` query to get multiple contents by id in one query.
* :zap: In Content type, `headline` field was renamed to `headlines` and now returns an array of strings.
* :zap: Entity names are now limited to 1024 characters.  Names will be truncated if they exceed the maximum length.
* :zap: In SummarizationTypes enum, `BULLET_POINTS` was renamed to `BULLETS`.
* :zap: In ProjectStorage type, `originalTotalSize` was renamed to `totalSize`, and `totalRenditionSize` field was added.  `totalSize` is the sum of the ingested source file sizes, and `totalRenditionSize` is the sum of the source file sizes and any derived rendition sizes.
* :zap: In ConversationStrategy type, `strategyType` was renamed to `type` for consistency with rest of data model.
* :zap: In Specification type, `optimizeSearchConversation` was removed, and now is handled by `OPTIMIZE_SEARCH` prompt strategy.

### Bugs Fixed

* GPLA-1725: Should ignore RSS.xml from web feed sitemap
* GPLA-1726: GPT-3.5 Turbo 16k LLM is adding "Citation #" to response
* GPLA-1698: Workflow not applied to link-crawled content
* GPLA-1692: Mismatched project storage total size, when some content has errored
* GPLA-1237: Add relevance threshold for semantic search


# October 30: Optimized conversation responses; added observable aliases; bug fixes

### New Features

* :bulb: Graphlit now supports 'aliases' of observable names, as the `alternateNames` property.  When an observed entity, such as Organization, is enriched, we store the original name and the enriched name as an alias.  For example, "OpenAI" may be enriched to "OpenAI, Inc.", and we store "OpenAI" as an alias, and update the name to "OpenAI, Inc.". 
* :bulb: Added `workflows` filter to ContentCriteriaInput type, for filtering content by workflow(s) when creating conversation.
* Optimized formatting of content sources into prompt context, for more accurate conversation responses.
* Optimized formatting of extracted text from Slack messages, for better knowledge retrieval.
* Updated text tokenizer for more accurate token counting.
* Upgraded Azure Text Analytics to latest preview API version.
* Authors found in RSS feeds are now stored as observations of Person entities.
* Added rate limiting for Reddit feeds.
* Added rate limiting for Wikipedia enrichment.
* Added support for reading Reddit post comments when reading Reddit feed.
* :zap: `EmbedFacets` has been renamed to `EnableFacets` in the conversation strategy.
* :zap: Removed extra `content` level in IngestionWorkflowStage type.  Now, the `if` property is of type IngestionContentFilter. 

### Bugs Fixed

* GPLA-1556: Better handling of very long user prompts.
* GPLA-1627: Optimized token budget for more accurate prompt completion.
* GPLA-1585: More accurate entity matching in Wikipedia entity enrichment.


# October 15: Support for Anthropic Claude models, Slack feeds and entity enrichment

### New Features

* :fire: Graphlit now supports Anthropic Claude and Anthropic Claude Instant large language models.
* :fire: Graphlit now supports Slack feeds, and will ingest Slack messages and linked file attachments from a Slack channel.  Note, this requires the creation of a Slack bot which has been added to the appropriate Slack channel.
* :bulb: Added support for entity enrichment to workflow object, which offers Diffbot, Wikipedia and Crunchbase enrichment of observed entities, such as Person, Organization and Place.
* :bulb: Added support for text extraction from images.  When using Azure Image Analytics for entity extraction, Graphlit will extract and store any identified text which then becomes searchable.
* Added `embedFacets` property to conversation strategy in specification object.
* Added `embedCitations` property to conversation strategy in specification object.  This makes content citations optional with the completed conversation message.
* Added GraphQL mutations for multi-delete of entities, such as `deleteCollections`, `deleteLabels`, or `deleteConversations`.
* Added GraphQL `deleteAllConversations` mutation to delete all conversations.
* Added support for automatically adding ingested content to one or more collections, via ingestion stage of workflow object.
* Added `specification` property to preparation workflow stage, which will be used to select the LLM for text summarization.
* Expanded the properties for observed entities, such as Person, Organization or Product.  Now supports a wider range of properties for entity enrichment.

### Bugs Fixed

* GPLA-1520: Unlimited conversation quota not assigned when upgrading project tier
* GPLA-1285: Entity enrichment not firing event, which can be sent to actions
* GPLA-1361: Web page left in ingested state, when URL not accessible.


# September 24: Support for YouTube feeds; added documentation; bug fixes

### New Features

* :fire: Graphlit now supports [YouTube feeds](https://docs.graphlit.dev/api-reference/graphql-data-model/feeds/create-youtube-feed), where you can ingest a set of YouTube videos, or an entire YouTube playlist or channel.   Note, we currently support only the ingestion of audio from YouTube videos, which gets transcribed and added to your conversational knowledge graph.

### New Documentation

* Added documentation for observable entities mutations and queries ([Label](https://docs.graphlit.dev/api-reference/graphql-data-model/observations/labels), [Category](https://docs.graphlit.dev/api-reference/graphql-data-model/observations/categories), [Person](https://docs.graphlit.dev/api-reference/graphql-data-model/observations/persons), [Organization](https://docs.graphlit.dev/api-reference/graphql-data-model/observations/organizations), [Place](https://docs.graphlit.dev/api-reference/graphql-data-model/observations/places), [Event](https://docs.graphlit.dev/api-reference/graphql-data-model/observations/events), [Product](https://docs.graphlit.dev/api-reference/graphql-data-model/observations/products), [Repo](https://docs.graphlit.dev/api-reference/graphql-data-model/observations/repos), [Software](https://docs.graphlit.dev/api-reference/graphql-data-model/observations/software)).
* Added documentation for using custom [Azure OpenAI](https://docs.graphlit.dev/api-reference/graphql-data-model/specifications/azure-openai) and [OpenAI ](https://docs.graphlit.dev/api-reference/graphql-data-model/specifications/openai)models with Specifications

### Bugs Fixed

* GPLA-1459: LLM prompt formatting was exceeding the token budget with long user prompts.
* GPLA-1445: Failed to ingest PDF from URL where filename in `Content-Disposition` header contained a backslash.


# September 20: Paid subscription plans; support for custom observed entities & Azure OpenAI GPT-4

### New Features

* :fire: Graphlit now supports paid [Hobby, Starter and Growth](https://www.graphlit.com/#pricing) tiers for projects, in addition to the existing Free tier.  Starting at $49/mo, plus $0.10/credit for usage, we now support higher quota based on your subscribed tier.   By providing a payment method for your organization in the Developer Portal, you can upgrade each project individually to the tier that fits your application's needs.
* :bulb: Added GraphQL mutations for the creation, update and deletion of observed entities (i.e. Person, Organization, Place, Product, Event, Label, Category).
* :bulb: Added new observed entity types to knowledge graph: **Repo** (i.e. Git repo), **Software**.
* :bulb: Added `searchType` and `numberSimilar` fields to Specification object for configuring semantic search in conversations.   In situations where the user prompt is limited in length, `HYBRID` search type can provide better semantic search results for the prompt context.
* :bulb: Added support for the Azure OpenAI GPT-4 model.
* Added support for project `quota` field.  Project quotas are based on the subscribed pricing tier.   Quota limits are now applied as content is ingested, and as feeds and conversations are created.
* Added `contentLimit` to conversation strategy object to limit the number of semantic search content results which are formatted into prompt context.
* Better relevance ranking on semantic search results when formatting prompt context in conversations.
* :information\_source: Free tier has updated quota: 1GB storage, 100 contents, 3 feeds and 10 conversations.
* :zap: Now using the [Deepgram Nova-2 audio transcription model,](https://deepgram.com/learn/nova-2-speech-to-text-api?utm_source=twitter\&utm_campaign=0923+Nova-2\&utm_medium=organic+social) which is 18% more accurate, and 5-40x faster.

### Bugs Fixed

* GPLA-1373: Failed to extract multiple text pages from DOCX without page breaks.  Now we support token-aware page chunking.
* GPLA-1377: Failed during semantic search with no content results, when prompting conversation.
* GPLA-1415: Failed when user prompt couldn't generate text embeddings.


# September 4: Workflow configuration; support for Notion feeds; document OCR

### New Features

* :fire: Added [**Workflow** entity](https://docs.graphlit.dev/api-reference/graphql-data-model/workflows) to data model for configuring stages of content workflow; can be assigned to Feed or with `ingestPage`, `ingestFile`, or `ingestText` mutations to control how content is ingested, prepared, extracted and enriched into the knowledge graph.
* :bulb: Added support for [Notion feeds](https://docs.graphlit.dev/api-reference/graphql-data-model/feeds/create-notion-feed): now can create feed to ingest files from Notion pages or databases (i.e. wikis).
* :bulb: Added support for API-created **Observation** entities, which allow for custom observations of observable entities (i.e. Person, Label) on Content.
* :bulb: Added support for [Azure AI Document Intelligence](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/overview?view=doc-intel-3.1.0) as an optional method for preparing PDF files, using OCR and advanced layout analysis.
* :bulb: Added summarization strategies, where content can be summarized into paragraphs, bullet points or headline.
* Added ability to assign [default Workflow and Specification to project](https://docs.graphlit.dev/api-reference/graphql-data-model/projects).
* Added more well-known link types, during link crawling, such as Discord, Airtable and TypeForm.
* :information\_source: Free/Hobby plan now has 5GB storage quota; any content ingested past that limit will be auto-deleted.
* :zap: **Actions** have been moved into Workflow entity.
* :zap: Link enrichment for Feeds has been moved into the Workflow enrichment stage, now called *link crawling*.  `ExcludeContentDomain` property has been reversed and is now called `IncludeContentDomain`.

### Bugs Fixed

* GPLA-1204: Failed to ingest content with backslash in name.
* GPLA-1276: Failed to ingest RSS posts which contained enclosure URI, but no post URI.


# August 17: Prepare for usage-based billing; append SAS tokens to URIs

### New Features

* :information\_source: Behind the scenes, Graphlit is preparing to launch usage-based billing.  This release put in place the infrastructure to track billable events.  Organizations now have a Stripe customer associated with them, and Graphlit projects are auto-subscribed to a Free/Hobby pricing plan.  In a future release, we will provide the ability to upgrade to a paid plan in the Graphlit Developer Portal.  Also, we will provide visualization of usage, on granular basis, in the Portal.
* :bulb: Content URIs now have Shared Access Signature (SAS) token appended, so they are accessible after query.  For example, content.transcriptUri will now be able to be downloaded or used directly in an application (until the SAS token expires).
* :bricks: Added more robustness for error handling and retries, especially for LLM APIs and audio transcription APIs.


# August 9: Support direct text, Markdown and HTML ingestion; new Specification LLM strategy

### New Features

* :bulb: Added `ingestText` mutation which supports direct **Content** ingestion of plain text, Markdown and HTML.  Now, if you have pre-scraped HTML or Markdown text, you can ingest it into Graphlit without reading from a URL.
* :bulb: Added **Specification** `strategy` property, which allows customization of the LLM context when prompting a conversation.  `ConversationStrategy` now provides `Windowed` and `Summarized` message histories, as well as configuration of the weight between existing conversation messages and Content text pages (or audio transcript segments) in the LLM context.
* :bulb: Added auto-summarization of extracted text and audio transcripts.  There is a new **Content** `summary` property where a list of summary bullet points can be found.  These summaries can be optionally included in the Conversation prompt context for more accurate LLM responses.
* :information\_source: Added `AzureOpenAIModels` and `OpenAIModels` types to Specification model properties to make it easier to specify the desired LLM.
* :information\_source: Renamed **ConversationMessage** `date` property to `timestamp`
* :sparkles: Refined the internal LLM prompts for providing content as part of Conversation context.  This provides for much clearer and accurate results from the LLM.


# August 3: New data model for Observations, new Category entity

### New Features

* :bulb: Revised data model for **Observations**, **Occurrences** and observables (i.e. Person, Organization).  Now after entity extraction, content will have one Observation for each observed entity, and a list of occurrences.  Occurrence now supports text, time and image occurrence types.  (Text: page index, time: start/end timestamp, image: bounding box)  Observations now have ObservableType and Observable fields, which specify the observed entity type and entity reference.
* :bulb: Added **Category** entity to GraphQL data model, which supports [PII](https://en.wikipedia.org/wiki/Personal_data) categories such as *Phone Number* or *Credit Card Number*.
* Added `probability` field to model properties, for the LLM's token probability.  (See [OpenAI documentation](https://platform.openai.com/docs/api-reference/chat/create#chat/create-top_p) for more detail.)
* Added `error` field to feeds.  If a feed fails to read from the data source, and is marked as `ERRORED` state, the `error` field will have the error description.
* Support reingestion of changed files from feeds.  For feeds, such as SharePoint or Web, where we can recognize that a file or page was updated, we will now reingest the content in-place.  Content will keep the same ID, and will restart the content workflow by re-downloading the updated content from the data source.   Existing observations will be deleted, and new observations will be created from the updated content.
* :information\_source: Ingestion of content is now idempotent, meaning if you ingest content again from the same URI, we will reingest the content in-place, while keeping the same ID.  (If we can recognize the content has not changed, such as by ETag, we will return the existing content object.)
* :information\_source: Changed GraphQL data type of SharePoint `tenantId`, `libraryId` and `siteId` to ID rather than String.
* :sparkles: Performance optimization of entity extraction, and the creation of observations.

### Bugs Fixed

* GPLA-1130: Only was extracting text from first column of PDF tables.
* GPLA-1140: Text from DOCX tables was not extracted properly.
* GPLA-1154: Audio content ingested from RSS feed was not deleted when feed was deleted.


# July 15: Support for SharePoint feeds, new Conversation features

### New Features

* :bulb: Added support for SharePoint feeds: now can create feed to ingest files from SharePoint document library (and optionally, folder within document library)
* :bulb: Added support for PII detection during entity extraction from text documents and audio transcripts: now we will create labels such as `PII: Social Security Number` automatically when PII is detected
* :bulb: Added support for developer's own OpenAI API keys and Azure OpenAI deployments in `Specifications`
* :information\_source:  Changed semantics of `deleteFeed` to delete the contents ingested by the feed; since contents are linked to feeds, now feeds can be disabled, while keeping the lineage to the feed, and if feeds are deleted, they will delete the linked contents, so we never lose the feed-to-content lineage
* Added GraphQL query for SharePoint consent URI, for registered Graphlit Platform Azure AD application
* Better handling of web sitemap indexes: now if a sitemap.xml contains a `sitemapindex` element, we will load all linked sitemaps for evaluating web pages to ingest from Web feed
* Added new GraphQL mutations for `openConversation`, `closeConversation` and `undoConversation`
* Added timestamps to Conversation messages
* Added new GraphQL mutations for `openCollection` and `closeCollection`
* Added more configuration for content search: now can specify `searchType` (KEYWORD, VECTOR, HYBRID) and `queryType` (SIMPLE, FULL - aka Lucene syntax)
* Better parsing of iTunes podcast metadata
* :zap: Renamed `listingLimit` field on feeds to `readLimit`
* :zap: Renamed `topK` to `numberSimilar` for content vector search type 
* :zap: Changed GraphQL feed properties: split out `azure` into `azureBlob` and `azureFile` properties
* :zap: Changed GraphQL specification properties: split out `openAI` into `openAI` and `azureOpenAI` properties
* :zap: Removed `count` fields on query results, and replaced with explicit `count{Entity}` queries, which support search and filtering.

### Bugs Fixed

* GPLA-1043: Reddit `readLimit` not taking effect: now the specified limit of Reddit posts will be leveraged for Reddit feeds
* GPLA-1064: Performance on entity extraction and observation creation for large PDFs was under expectations: now able to build knowledge graph from large PDFs much faster (4x speed improvement)
* GPLA-1053: If rendition generation errored during content workflow, the content was not properly marked as errored
* GPLA-1102: Large Web sitemaps were slow to load; rewrote sitemap index handling, and now can process sitemaps with 150K+ entries in seconds.