FIDELIS Week 0

We are looking for LLM tools that can be deployed on Stretch Robot solely using CPU computation power. We found Ollama is extremely fast running on CPU and it can even runs on a Raspberry Pi.

We also need the LLM application to have the Retrieval-augmented generation(RAG) ability since our newly generated FSMs are based on some already existed FSMs. We found LangChain could be a good companion with Ollama.

Install Ollama + LangChain

conda create -n localfsm python=3.11
conda activate localfsm
 
# download code here
# https://github.com/ollama/ollama/tree/main/examples/langchain-python-rag-document
 
# install Ollama
# https://ollama.com/download
 
# install LangChain and other req
pip install -r requirements.txt
 
python main.py

The next step is looking for some FSMs data as examples for this LLMs to refer. We plan to use two types of data:

1. Some well-written FSMs by human, and they have been verified on physical robots.
2. Use larger LM, like GPT4 or Claude 3 to generated some FSMs

FSM knowledge:

• https://www.cs.cmu.edu/afs/cs/academic/class/15494-s17/lectures/fsm/fsm.pdf

Navigation: Navigate to Rearrangement

• Help me pick an apple
• Help me pick an apple and put it on the table
• go to the table
• Guessing number machine
• I spilled the water on the table, help me clean the table
• I have three cubes in red, yellow, and green, help me stack them in them yellow, green, red

1. read from PDF
2. customized. format

{
  "name": "Finite State Machine Name",
  "initialState": "State1",
  "states": [
    {
      "name": "State1",
      "transitions": [
        {
          "to": "State2",
          "on": "Event1"
        },
        {
          "to": "State3",
          "on": "Event2"
        }
      ]
    },
    {
      "name": "State2",
      "transitions": [
        {
          "to": "State1",
          "on": "Event3"
        }
      ]
    },
    {
      "name": "State3",
      "transitions": [
        {
          "to": "State1",
          "on": "Event4"
        }
      ]
    }
  ]
}

Synthetic data generation

I also tried to use Modelfile in Ollama, the results were not bad.

ollama create fsm -f fsm.Modelfile
ollama run fsm:latest