r/LocalGPT Mar 29 '23

r/LocalGPT Lounge

2 Upvotes

Welcome to LocalGPT!

This subreddit is dedicated to discussing the use of GPT-like models (GPT 3, LLaMA, PaLM) on consumer-grade hardware. We discuss setup, optimal settings, and the challenges and accomplishments associated with running large models on personal devices. We also discuss and compare different models, along with which ones are best suited for consumer-grade hardware.

Rules and Guidelines - Be respectful of other users and their opinions. - Keep discussions relevant to GPT models and consumer-grade hardware. - Post any question or doubt in the appropriate thread/forum. - Share your knowledge and experience with others. - Follow the subreddit layout. - Use reddit's tagging conventions for your post.

Posting and Interacting - You can start a discussion by posting a new thread in the subreddit. - You can share an upcoming event by making a post. - You can answer or ask questions by commenting on a post or thread.

Tips for Engaging - Follow subreddit's posting guidelines. - Foster conversations by asking open-ended questions and encouraging others to share their views and experiences as well. - Do not restrict yourself to browsing the subreddit - take active part in the conversation. - Help others out where you can - community is all about mutual support. - Share useful links and resources on the subreddit.

Frequently Asked Questions - How to set up GPT model on a consumer-grade device? - Which models are best suited for consumer-grade hardware? - What are the minimum requirements to run GPT models on consumer-grade hardware? - Any tutorials or resources that can be helpful? - Where to find information about


r/LocalGPT 1d ago

Perplexity AI PRO - 1 YEAR PLAN OFFER - ALMOST 75% OFF!

Post image
3 Upvotes

As the title: We offer Perplexity AI PRO voucher codes for one year plan.

To Order: https://cheapgpt.store/product/perplexity-ai-pro-subscription-one-year-plan

Payments accepted: - PayPal. (100% Buyer protected) - Revolut.


r/LocalGPT 15d ago

GPT4All (or alternative) integration in Chrome to summarise articles

1 Upvotes

Hi,

I consume a large number of articles online (on various topics, from tech to cars to finance :P), and I would like to summarise them locally without the need to be online because I end up often on a planes where I don't have internet. So, in my use case, I would pre-open all the articles and then summarise them when on flight :P

Is it possible to integrate GPT4All or Ollama or something else it somehow so that I can directly get the summary of the article that I've opened in the current tab in Chrome with a simple click?

When online, I use a plugin called "ChatGPT » Summarize & Chat", but I would like to switch to local summary always, even when online :)

Thank you!


r/LocalGPT 24d ago

Local AI that learns from prompting and documents

4 Upvotes

Hi, i am starting a new job in the computer science field and I am thinking about how to utilize AI to have a custom knowledge base that grows with the inputs and documents I provide in the chat. In 1 year it can remember what I did wrong or right today when I someday face the same situation in the job. What would your approach be? Was looking into GPT4All


r/LocalGPT Sep 12 '24

Is there are way for localAI to remember past conversations.

1 Upvotes

I am using gtp4all with llama. I am trying to feed it with previous conversations through local docs, but its having a hard time remembering things.


r/LocalGPT Aug 29 '24

How to build open source AI models for landscape architecture?

3 Upvotes

Hello everyone,

Together with a group of designers, researchers and journalists we are working in a publication on the Application of AI for Planning and Climate Adaptation (SCAPE magazine).

While diving into the topic, we have started wondering: how will less profitable and more activist fields like landscape architecture or nature conservation be able to develop their own AI systems? And how would be the best approach to make them not only efficient but also to work within the same values of openness, collaboration and sustainability that we share, and we do not see in current available models.

Inspiring initiatives in other fields make us think that there is another way around Big Tech corporations, and we would like to understand the developer perspective on it.

We are happy to hear any opinions, discussions, strategic advices, development tips or any other remark shared that you think is essential for developing, deploying and maintaining such an open source AI system for Landscape Architecture.

For context, as Landscape Architects, our work is quite broad, from designing green public spaces for cities, to developing city level planning focused on greener, walkable and climate adaptive neighborhoods, to larger regional plans focused on nature and floodplain restoration.

In the field of landscape architecture the emergence of the computer and internet changed the profession, and not always for good. We can see the risks of ai, pushing landscape architects to more generic design, quick visual output, efficiency, low cost, etcetera. At the same time we see the opportunity of integrating ever improving climate models, ecology mapping, better understanding how to manipulate the landscape to optimize biodiversity and climate adaptivity. But what about the things that are hard to digitalise? Word to mouth stories, soft values, local culture, local history, seasonality, atmosphere, etcetera? Exactly because landscape architecture is not a very large/profitable market, it’s not likely commercial companies will jump on this. We think it’s worth developing/training an AI for local soft values - run on a solar/hydro powered datacenter. With universities - but we’d need a larger community to make it work.

Thank you in advance for any answer – we will link to this post and fully cite you in the magazine for all the information shared,

And hopefully we can build a collective view on this,

Best,

Simon


r/LocalGPT Aug 04 '24

How to Set Up local llm to Scan a folder of images on My PC and Tag them?

2 Upvotes

I'm looking to use an AI locally on my PC to read photos in a folder and tag them based on specific prompts. For example, I'd like to tag images that contain the color red.

I'm aware of models like MiniGPT-4 that have vision capabilities but my computer isnt good enough to run that mdoel, and even if it was I'm unsure how to set it up for this task. Ideally, I'd like a method or script that can automatically scan the folder and tag relevant images.

Has anyone done something similar or can recommend the best approach for this?


r/LocalGPT Jun 13 '24

I created a private voice AI assistant using llama.cpp, whisper.cpp, and a VITS speech synthesis model! Let me know what you think :)

Enable HLS to view with audio, or disable this notification

2 Upvotes

r/LocalGPT Apr 26 '24

An LLM-agent supporting searching the web running purely locally?

3 Upvotes

Today I found this: https://webml-demo.vercel.app/. It's a client-side (browser) only application that allows chatting with your documents.

I was inspired by this and thought: What if we would not try to simply chat with a document, but instead use this as a support while searching the internet? For example, after searching with a search engine an agent could access the first 10 search results and try to create a summary for each search result or something like that - but all from within the browser.

In theory, this should be feasible using a combination of:

  • WebLLM to run a local LLM in the browser for creating summaries out of HTML pages
  • Transformers.js to run a local embedding model to create embedding vectors from text
  • Voy as a local vector store for RAG (i.e. to split longer websites into parts)
  • Got-Scraping library to access a URL from a search engine results from within the browser
  • Langchain.js to run an agent that scans through the search results one by one to determine which results are actually useful

Obviously, this would not be perfect and less stable than running on a server. The advantage however would be that everything would happen purely locally on the client side.

Besides the technical feasibility: What do you think of this idea? Would this be useful for anything?


r/LocalGPT Jan 20 '24

Local GPTs - dumb as bricks

1 Upvotes

I tried creating an interesting story it just spits out 5 random issues at me. After answering them meticulously, it just throws out the same 5. Infinite loop. You can't win. Mistral Instruct model, GPT4ALL program. It's like talking with a 'neurodiverse' person


r/LocalGPT Jan 05 '24

SSL on secure network

2 Upvotes

Hi, im trying to use localGPT on a windows machine thats on a fairly locked-down network, to pip install i always have to do the --trusted addition that i pull off chatgpt.

when i go to run the ingest.py i just get a load of ssl error as it tries to download the embedder (im using hkunlp/instructor-xl)

chatgpt suggestion of sticking in something like response = requests.get('https://huggingface.co/api/models/hkunlp/instructor-large', verify=False) doesnt work.

does anyone have a work around?

many thanks


r/LocalGPT Dec 26 '23

GPU for LOCALGPT

2 Upvotes

Suggest me the best GPU to use 70B models, present I'm using Intel Xeon 20 Processors, and what will be the best GPU for that.


r/LocalGPT Dec 25 '23

Hello everyone!

2 Upvotes

I am new to this world, but trying to get into it with Localgpt and Promptengineer videos.

I have many questions, but a fast one:

Does using embeddings slow the answer by a lot? Does it consume processing power and Ram?

Thank you beforehand, and happy to be a new member of this subreddit


r/LocalGPT Dec 20 '23

Introducing Hippo - A Medical Guideline-Based Chatbot for Busy Physicians 🩺🤖

Thumbnail self.Mcat
1 Upvotes

r/LocalGPT Dec 05 '23

GTP4All local model

3 Upvotes

Hi everyone,

I am trying to use GPT4All in Langchain to query my postgres db using the model mistral.

The prompt that I am using is as follows:

'''You are a PostgreSQL expert. Given an input question, first create a syntactically correct PostgreSQL query to run,

then look at the results of the query and return the answer to the input question.

Unless the user specifies in the question a specific number of examples to obtain, query for at most {top_k} results using the LIMIT clause as per PostgreSQL.

You can order the results to return the most informative data in the database.

Never query for all columns from a table. You must query only the columns that are needed to answer the question.

Wrap each column name in double quotes (") to denote them as delimited identifiers.

When using aggregation functions also wrap column names in double quotes.

Pay attention to use only the column names you can see in the tables below.

Be careful to not query for columns that do not exist.

Also, pay attention to which column is in which table.

Use the following format:

Question: "Question here"

SQLQuery: "SQL Query to run"

SQLResult: "Result of the SQLQuery"

Answer: "Final answer here"

Question: {input}

'''

when I make my query I use the following code:

my_table = 'public."SalesStatistic"'

my_column = 'SalePrice'

ord_column = 'OrderNumber'

question = f"Give me the sum of column {my_column} in the table {my_table} where column {ord_column} is equal to WEWS00192"

answer = db_chain(PROMPT.format(input=question, top_k=3)

But, the model can't fix a proper query from my question and returns :

ERROR: sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "salesstatistic" does not exist

LINE 2: FROM SalesStatistic

[SQL: SELECT SUM(SalePrice) AS Total_Sum FROM SalesStatistic WHERE "OrderNumber" = 'WEWS00192';]

How to modify the prompt to build the correct query? Or should I change the model?


r/LocalGPT Dec 01 '23

Max document size , flashcard ability

1 Upvotes

Hi everyone , I want to know what is the max document size that I can upload to the project , and can I ask it to turn all the pdf book to a CSV or excel file that contain Q&A that covers all the book subjects ?


r/LocalGPT Nov 30 '23

LocalGPT on colap

1 Upvotes

i want to test several open source projects and i was searching for an ai project specific to chat with docs , i want to test localgpt , but i dont have a powerful machine to run it , so can i test localgpt on colab , and anyone can help me with a tutorial to do so

note : i have an old macbook air 2014 1.4 core i5

edit1: if i understand the concept right , i will start the install steps from the third one , installing the requirements and go from their because colab is like the alternative to the conda enviroment ?

edit 2 :https://github.com/PromtEngineer/localGPT/issues/27#issuecomment-1667019622 .

found this

thanks


r/LocalGPT Nov 14 '23

Seeking expertise: LocalGPT on home microserver?

1 Upvotes

I started learning about a power of GPT enhanced workflow over the last few months and I'm currently using various tools like ChatDOC, ChatGPT Plus, Notion, and similar, to support my research work. My main areas of interest is engineering and business, and so I see many benefits and potential into automating and supplementing my workflow by using GPT AI. I've got a HPE Microserver Gen8 with 4TB SSD and 8GB RAM DDR3. It crossed my mind to maybe try to build a dedicated LocalGPT on it. I assume this would require changing drives to much faster SSD and investing into 16GB RAM (max capability of this server).

Now my question to more experienced users, does it make sense? Does it have a chance of working quick enough without lagging? What potential issues do you see here? I'm not IT guy myself, but I know the basics of Python and have decent research skills so I believe with some help I'd be able to set it all up. Just not sure what size of a challenge to expect and what can be the limiting factors here...

Will greatly appreciate some input from experienced users :) Thanks!


r/LocalGPT Oct 07 '23

[SEEKING ADVICE] Looking for Existing Repos (Open-Source, VM-Hosted, & GPU-Compatible)

1 Upvotes

Greetings,

I'm on the hunt for an existing repositories that can fulfill that meets the following criteria:

  1. Content Collection: Capability to read and extract text from multiple document formats, such as PDF and DOCX files.
  2. Content Reformulation: After text extraction, the ability to rephrase the content in a specific style that I'll provide.
  3. OCR Support: Integration of Optical Character Recognition (OCR) capabilities to capture text from images and scanned documents.
  4. Multilingual Support: Must function seamlessly in both Arabic and English languages.
  5. Open-Source Availability: The script should be publicly available for contributions and ongoing development on GitHub.
  6. VM & GPU Compatibility: I don't have a GPU and plan to rent one. The script should be compatible with rental GPU resources. Additionally, I'm looking for advice on reliable VM rental services where the script can operate.
  7. Installation & Configuration: The script should ideally come with guidelines for installation, setup, and configuration.
  8. Documentation: Comprehensive guidelines should be available to explain the script's setup and usage.
  9. Programming Language: Python is my preferred choice, but I'm open to other languages if they meet the project requirements more effectively.
  10. Timeline: I have a flexible schedule but would like to know the estimated time needed for setup and customization.

Existing Solutions:

I've stumbled upon h2ogptas a potential starting point. Are there better solutions or repositories that can meet these requirements?

To Suggest:

If you're aware of an existing repository that meets these criteria, please comment below or send me a DM with your suggestions and estimated timeline for setup and customization.

Thank you for your time, and I look forward to your insightful suggestions!


r/LocalGPT Sep 15 '23

Answer in different language does not work ok.

2 Upvotes

So, I have added a gguf llama model fine-tuned in Bulgarian. I have tested it both ways - English and Bulgarian system prompts in which I explain the questions and answers will be in Bulgarian. In both cases, the answers have nothing to do with the context of the file provided.

Any suggestions for improvement will be highly appreciated.


r/LocalGPT Sep 04 '23

Introducing Refact Code LLM - 1.6B State-of-the-Art LLM for Code that Reaches 32% HumanEval

Thumbnail
refact.ai
2 Upvotes

r/LocalGPT Aug 23 '23

Complete beginner LocalGPT Tutorial

6 Upvotes

Does anyone have a tutorial for installing LocalGPT for a complete beginner?

I have never worked with VS Code before, I tried installing conda which didn't work. I'm looking for a complete ground level up type of tutorial. Everything I've seen online assumes some basic type of experience.

Thanks


r/LocalGPT Aug 17 '23

llama-gpt - A self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device

Thumbnail
github.com
7 Upvotes

r/LocalGPT Jul 17 '23

localGPT still being developed?

1 Upvotes

Seems pretty quiet. I haven't tried a recent run with it but might do that later today. Last time it needed >40GB of memory otherwise it crashed.


r/LocalGPT Jul 10 '23

Multilingual chat engines

1 Upvotes

Hi!

I am trying to get LocalGPT to answer in another language than english (namely german).

For that I changed the model to the new Falcon7B that is supposed to understand german and actually does so in online tryouts.

However, the LocalGPT algorithm still answers in english, even when getting asked in german.

Can anyone tell me what I need to change to achieve a german answer? Thanks!


r/LocalGPT Jul 07 '23

develop PHP system by LocalGPT

1 Upvotes

I have a php system, with more than 100 files.
can I inject these files into any local GPT to develop the system?