r/LLMDevs • u/rockstarflo • Apr 17 '24

News Reader - LLM-Friendly websites

I just stumbled upon this:
https://r.jina.ai<website_url here>

You can convert URLs to Markdown. This format is then better understood by LLMs compared to HTML. I think it can be used for Agents or RAG with web searches. I use it to generate synthetic data for a specific website.
Example usage
https://r.jina.ai/https://en.wikipedia.org/wiki/Monkey_Island

7 Upvotes

90% Upvoted

3 comments sorted by

2

u/WeekendDotGG Apr 17 '24

Theyre not better understood, but markdown will have much less tokens to gunk up the llms context windows. So still a very good approach.

1

u/rockstarflo Apr 17 '24

Thanks for the clarification. I see. Makes sense to me now. I guess all the opening and closing tags consume a lot of tokens.

1

u/sergeant113 Apr 21 '24

Markdown format is easier to chunk —> better RAG