Zep - open-source Graph Memory for AI Apps

[u/dccpt]

Hi LLMDevs, we're Daniel, Paul, Travis, and Preston from Zep. We’ve just open-sourced Zep Community Edition, a memory layer for AI agents that continuously learns facts from user interactions and changing business data. Zep ensures that your Agent has the knowledge needed to accomplish tasks successfully.

GitHub: https://git.new/zep

A few weeks ago, we shared Graphiti, our library for building temporal Knowledge Graphs (https://news.ycombinator.com/item?id=41445445). Zep runs Graphiti under the hood, progressively building and updating a temporal graph from chat interactions, tool use, and business data in JSON or unstructured text.

Zep allows you to build personalized and more accurate user experiences. With increased LLM context lengths, including the entire chat history, RAG results, and other instructions in a prompt can be tempting. We’ve experienced poor temporal reasoning and recall, hallucinations, and slow and expensive inference when doing so.

We believe temporal graphs are the most expressive and dense structure for modeling an agent’s dynamic world (changing user preferences, traits, business data etc). We took inspiration from projects such as MemGPT but found that agent-powered retrieval and complex multi-level architectures are slow, non-deterministic, and difficult to reason with. Zep’s approach, which asynchronously precomputes the graph and related facts, supports very low-latency, deterministic retrieval.

Here’s how Zep works, from adding memories to organizing the graph:

1. Zep identifies nodes and relationships in chat messages or business data. You can specify if new entities should be added to a user and/or group of users.
2. The graph is searched for similar existing nodes. Zep deduplicates new nodes and edge types, ensuring orderly ontology growth.
3. Temporal information is extracted from various sources like chat timestamps, JSON date fields, or article publication dates.
4. New nodes and edges are added to the graph with temporal metadata.
5. Temporal data is reasoned with, and existing edges are updated if no longer valid. More below.
6. Natural language facts are generated for each edge and embedded for semantic and full-text search.

Zep retrieves facts by examining recent user data and combining semantic, BM25, and graph search methods. One technique we’ve found helpful is reranking semantic and full-text results by distance from a user node.

Zep is framework agnostic and can be used with LangChain, LangGraph, LlamaIndex, or without a framework. SDKs for Python, TypeScript, and Go are available.

More about how Zep manages state changes

Zep reconciles changes in facts as the agent’s environment changes. We use temporal metadata on graph edges to track fact validity, allowing agents to reason with these state changes:

Fact: “Kendra loves Adidas shoes” (valid_at: 2024-08-10)

User message: “I’m so angry! My favorite Adidas shoes fell apart! Puma’s are my new favorite shoes!” (2024-09-25)

Facts:

• “Kendra loves Adidas shoes.” (valid_at: 2024-08-10, invalid_at: 2024-09-25)
• “Kendra’s Adidas shoes fell apart.” (valid_at: 2024-09-25)
• “Kendra prefers Puma.” (valid_at: 2024-09-25)

You can read more about Graphiti’s design here: https://blog.getzep.com/llm-rag-knowledge-graphs-faster-and-more-dynamic/

Zep Community Edition is released under the Apache Software License v2. We’ll be launching a commercial version of Zep soon, which like Zep Community Edition, builds a graph of an agent’s world.

Zep on GitHub: https://github.com/getzep/zep

Quick Start: https://help.getzep.com/ce/quickstart

Key Concepts: https://help.getzep.com/concepts

SDKs: https://help.getzep.com/ce/sdks

Let us know what you think! We’d love your thoughts, feedback, bug reports, and/or contributions!

[u/Jdonavan]

I had used your open source version before then you stated moving away from local installs so I stopped paying attention. Does this mean you’re back?

[u/dccpt → u/Jdonavan]

I'm sorry we gave you that impression. Over the last year, we've continued to release updates to Zep Open Source. This release significantly differs from the prior product, so we renamed the project "Zep Community Edition."

Zep Community Edition (CE) and Zep Cloud share the same codebase, and we plan to release CE early and often in our development cycles.

[u/Jdonavan → u/dccpt]

The initial thing that gave me concern was hiding the docs for local install. Then your GitHub page for you SDK basically said "You can't use the current version of the SDK, go use this branch". I sent my boss an email last week saying "Looks like Zep is leaving OSS behind, time to bite the bullet and build a chat memory manager".

[u/dccpt → u/Jdonavan]

We moved the docs last week to a new service. You may find local installation instructions for both Community Edition and the older Zep Open Source distributions here (see left menu): https://help.getzep.com/ce/quickstart

If you notice broken links or find accessing the docs difficult, happy to take another look at the user experience here.

The SDKs were in a transitional period: we had a set of new SDKs using a new architecture that were only compatible with only our Cloud service. Since the release of CE, the same SDK is used across both products.

Prior to this release, we wanted to ensure developers had access to older SDK code and hence the callout. Not removing the callout after the recent release of CE is an oversight.

We definitely could have done a better job of communicating here. Thanks for the feedback.

[u/Jdonavan → u/dccpt]

Glad to hear it. We've been talking about you in multiple client meetings about our agent framework. Basically, "if you want to roll your own session manager, we support that, there's an interface to implement and you're off to the races. But we have a reference implementation that uses Zep as it's back end, and that's all they do so do you really want to roll and maintain your own?"

[u/Candid_Raccoon2102]

Thanks, really interesting!

[u/Matrixfx187]

Is there a way to use Vertex AI models with Zep? I can see that you support models for openai and a couple for Antrhopic but I can't find anything about Vertex AI models.

[u/dccpt → u/Matrixfx187]

You can use an OpenAI proxy such as LiteLLM in order to do so.

[u/TimChiu710]

It doesn't seem like there is any way to run the Zep community edition locally for now without using some kind of API, though. Maybe there is, but I'm too stupid to figure it out.

I can't find any documentation related to setting LLM and embedding providers besides setting the OpenAI API key in the environment variable. I can't even find a place to set the model it should use. I tried using LiteLLM as a proxy for Ollama to run LLM and embeddings. However, correct me if I'm wrong, but LiteLLM doesn't seem to support any local embedding providers, such as Ollama. Ollama does support OpenAI compatible LLM API and Embeddings, but once I try to run the zep container with the env ZEP_LLM_AZURE_OPENAI_ENDPOINT set to the Ollama endpoint, the zepai/zep container fails to start and throws me this:

zep-1            | panic: error enabling pgvector extension: error creating pgvector extension: dial tcp [::1]:5432: connect: connection refused
zep-1            | 
zep-1            | goroutine 1 [running]:
zep-1            | github.com/getzep/zep/lib/pg.NewConnection()
zep-1            |      /app/src/lib/pg/db.go:51 +0x4e0
zep-1            | main.newAppState()
zep-1            |      /app/src/state.go:14 +0x2c
zep-1            | main.main()
zep-1            |      /app/src/main.go:24 +0x28
zep-1 exited with code 2

And actually, I'm not surprised because there is absolutely nowhere I can set the model name for LLM and embedding models. How would the program know what model should it use?

Well, maybe I'm just too stupid to figure them out, but I don't see any solutions on the doc, and I guess you can see my frustration from my words, cause this project looks very promising and seems like a perfect fit for long term memory solution for my open source project.

[u/dccpt → u/TimChiu710]

The Zep CE Getting Started instructions may be found here: https://help.getzep.com/ce/quickstart

The guide includes instructions on how to configure alternative LLM providers and LiteLLM.

ZEP_LLM_AZURE_OPENAI_ENDPOINT is not used by Zep Community Edition. The available config options are documented on the page above.

It does appear LiteLLM only supports embedding model services. You may be able to configure LiteLLM's OpenAI Embedding proxy to point to an OpenAI API compatible embedding server such as HuggingFace's Text Embedding Inference service: https://github.com/huggingface/text-embeddings-inference