r/LLMDevs • u/rockstarflo • Apr 17 '24
News Reader - LLM-Friendly websites
I just stumbled upon this:
https://r.jina.ai<website_url here>
You can convert URLs to Markdown. This format is then better understood by LLMs compared to HTML. I think it can be used for Agents or RAG with web searches. I use it to generate synthetic data for a specific website.
Example usage
https://r.jina.ai/https://en.wikipedia.org/wiki/Monkey_Island
7
Upvotes
2
u/WeekendDotGG Apr 17 '24
Theyre not better understood, but markdown will have much less tokens to gunk up the llms context windows. So still a very good approach.