Links

August 3: New data model for Observations, new Category entity

New Features

  • 💡
    Revised data model for Observations, Occurrences and observables (i.e. Person, Organization). Now after entity extraction, content will have one Observation for each observed entity, and a list of occurrences. Occurrence now supports text, time and image occurrence types. (Text: page index, time: start/end timestamp, image: bounding box) Observations now have ObservableType and Observable fields, which specify the observed entity type and entity reference.
  • 💡
    Added Category entity to GraphQL data model, which supports PII categories such as Phone Number or Credit Card Number.
  • Added probability field to model properties, for the LLM's token probability. (See OpenAI documentation for more detail.)
  • Added error field to feeds. If a feed fails to read from the data source, and is marked as ERRORED state, the error field will have the error description.
  • Support reingestion of changed files from feeds. For feeds, such as SharePoint or Web, where we can recognize that a file or page was updated, we will now reingest the content in-place. Content will keep the same ID, and will restart the content workflow by re-downloading the updated content from the data source. Existing observations will be deleted, and new observations will be created from the updated content.
  • ℹ️
    Ingestion of content is now idempotent, meaning if you ingest content again from the same URI, we will reingest the content in-place, while keeping the same ID. (If we can recognize the content has not changed, such as by ETag, we will return the existing content object.)
  • ℹ️
    Changed GraphQL data type of SharePoint tenantId, libraryId and siteId to ID rather than String.
  • Performance optimization of entity extraction, and the creation of observations.

Bugs Fixed

  • GPLA-1130: Only was extracting text from first column of PDF tables.
  • GPLA-1140: Text from DOCX tables was not extracted properly.
  • GPLA-1154: Audio content ingested from RSS feed was not deleted when feed was deleted.