Furhat Realtime API: Complete Setup and Workflow Guide

What this guide is

This guide takes you from nothing to a Furhat robot that holds a real spoken conversation. It covers the Furhat Realtime API, the WebSocket API shown in the Furhat demo video. The API streams audio, video, and robot commands back and forth in real time between the robot and your Python code.

Whoever reads this page gets: the full setup, the starter code files, and an in-depth explanation of all five workflows from the video (connecting, LLM dialogue with lip-sync, local models, computer vision, and telepresence).

What you need before you start

1. The Furhat SDK installed and running. It includes a free virtual Furhat robot. (Your Furhat guide covers the install.)
2. Python 3.10 or newer. Check with python --version in a terminal.
3. An OpenAI API key, only needed for the chatbot script (02_chatbot.py).
4. Realtime API access. The Realtime API is part of the FurhatAI subscription, so the lab's physical Furhat needs that enabled. The virtual robot in the SDK works for development.

Setup, step by step (do this once)

Open a terminal in the folder that holds the starter files (the files folder linked at the bottom of this page), then follow each step in order.

python -m venv venv

.\venv\Scripts\activate

source venv/bin/activate

pip install -r requirements.txt

copy files\.env.example .env

cp files/.env.example .env

OPENAI_API_KEY=put-your-key-here
FURHAT_HOST=127.0.0.1
FURHAT_AUTH_KEY=

Run it

python 01_hello_world.py

python 02_chatbot.py

python 01_hello_world.py --host=192.168.0.52 --auth_key=mykey123

Part 1: Connecting to the robot (in depth)

The idea. The Realtime API is a WebSocket server that runs on the robot (or on the virtual robot inside the SDK). Your Python script opens a connection to it, and from then on the two sides exchange messages in real time: your script sends commands (speak, move the face, change the light), and the robot sends back events (I heard speech, here is a camera frame, the user moved).

How the connection works, step by step

The code (verified)

from furhat_realtime_api import FurhatClient

furhat = FurhatClient("127.0.0.1")   # or the robot's IP
# furhat = FurhatClient("192.168.1.50", "YOUR_AUTH_KEY")  # if a key is needed
furhat.connect()
furhat.request_speak_text("Hello, I am Furhat.")
furhat.disconnect()

Sync vs async client

The package ships two clients. FurhatClient is synchronous: each call blocks until the robot answers, which makes simple scripts easy to read. AsyncFurhatClient is asynchronous: calls return immediately and events arrive as they happen, which is what you want when several streams (audio in, audio out, camera) must run at the same time. Start with the sync client; move to async when you add streaming.

Part 2: LLM dialogue and real-time lip-sync (in depth)

The idea. A talking robot needs three jobs done: hear the user (speech-to-text), think of an answer (the LLM), and speak it (text-to-speech). The Realtime API lets you arrange these jobs however you like, and the robot's mouth moves in sync with whatever audio comes out.

Option A: let the robot do the hearing (simplest, verified)

The robot has built-in speech recognition. request_listen_start() waits until the user finishes speaking and returns the recognized text as a plain string. Your loop is then: listen, send the text to the LLM, speak the answer.

furhat.request_speak_text("What would you like to talk about?")
while True:
    user_text = furhat.request_listen_start()  # blocks, returns text
    if not user_text.strip():
        furhat.request_speak_text("Sorry, I did not catch that.")
        continue
    if user_text.lower() in {"goodbye", "bye"}:
        furhat.request_speak_text("Goodbye!")
        break
    reply = ask_llm(user_text)        # your OpenAI / local LLM call
    furhat.request_speak_text(reply)  # robot speaks, lips move automatically

You can tune the listening behavior (all verified parameters):

furhat.request_listen_config(languages=["en-US"])  # recognition language
furhat.request_listen_start(
    partial=False,           # True also returns in-progress partial results
    no_speech_timeout=8.0,   # give up after 8 seconds of silence
    end_speech_timeout=1.0,  # user is done after 1 second of silence
)

This is exactly what 02_chatbot.py does, with OpenAI as the LLM and the full conversation history passed along so the robot remembers context.

Option B: stream your own audio with lip-sync (verified)

When your text-to-speech runs outside the robot (a cloud voice or a local model), you stream the audio file and the face lip-syncs by itself. This is the mechanism behind the video's karaoke demo: any audio you send plays through the robot's speaker with only about 0.2 seconds of delay, and the mouth matches it.

furhat.request_speak_audio(
    url="https://your-server/audio.wav",  # any audio URL the robot can reach
    text="transcript for context",        # optional
    lipsync=True,   # mouth moves with the audio
    abort=False,    # True cuts off whatever is currently playing
    wait=False,
)

Karaoke: one way to build it (your design, not shown in the video)

Simpler alternative: stream the full mixed song the same way. The lip-sync still follows the vocal frequencies, and you skip the splitting step entirely.

Part 3: Local models, no cloud (in depth)

The idea. Instead of sending the user's voice to cloud services, you run everything on your own computer: speech-to-text, the LLM, and text-to-speech. Nothing leaves the lab. This is exactly what the video demonstrates: the whole pipeline on a MacBook next to the robot.

The key architectural point. In the video's diagram, user speech flows from the robot to the computer. You use the robot's microphone, not your laptop's. The robot streams raw audio to your script, your script runs its models, and the answer streams back to the robot's speaker.

Robot mic --(request.audio.start)--> your computer --(request.speak.audio)--> Robot speaker
                                     |- STT: faster-whisper (local)
                                     |- LLM: Ollama, e.g. Llama 3 or Mistral
                                     |- TTS: Piper or Coqui (local WAV)

Step by step

await furhat.request_audio_start(sample_rate=16000, microphone=True, speaker=False)
# audio chunks arrive as response.audio.data events
# ...
await furhat.request_audio_stop()

furhat.request_speak_audio(url="http://<your-pc>/reply.wav", lipsync=True)

Part 4: Computer vision (in depth)

The idea. The robot's camera feed streams out through the API into your Python code, where you run real vision models (the video uses YOLO object detection on a GPU). What the model sees then drives what the robot does: look at the person, react, change its light.

Step 1: get camera frames (verified, corrected)

There is no request.vision.stream event. The real API is:

frame = furhat.request_camera_once()  # one frame, returned directly

# Continuous feed (async client): frames arrive as response.camera.data
await furhat.request_camera_start()
# ... process frames as they arrive ...
await furhat.request_camera_stop()

Similarly, request_users_once() (or the request.users.start stream) gives you tracked-user data: who is in front of the robot and where they are.

Step 2: run YOLO (model name corrected)

The video's dashboard runs yolov8m.pt (YOLOv8-medium) on a CUDA GPU, with the confidence threshold set to 0.25 and classes like person, bicycle, car, and motorcycle selected.

from ultralytics import YOLO

model = YOLO("yolov8m.pt")  # same model family as the video's demo

def process_frame(frame):
    results = model(frame, conf=0.25)
    for r in results:
        for box in r.boxes:
            label = model.names[int(box.cls)]  # e.g. "person"
            x1, y1, x2, y2 = box.xyxy[0]       # where it is in the frame

Step 3: make the robot react (verified)

furhat.request_attend_user(user_id="closest")  # look at the nearest person
furhat.request_attend_location(x=0.0, y=0.0, z=1.0)  # or look at a 3D point
furhat.request_gesture_start("Nod")                  # react with a gesture
furhat.request_led_set("green")                      # signal state with the neck light

Example behavior: when a "person" box appears, call request_attend_user("closest") so the robot turns to face them, then greet them with request_speak_text(...).

Part 5: Telepresence (in depth)

The idea. A remote human operator "becomes" the robot: their voice comes out of its speaker, their facial expressions and gaze drive its face, and they see and hear the room through its camera and microphones. The video shows this working from a VR headset (C# client) and from a plain web browser (JavaScript client). Any language with WebSocket support can do it.

The four streams (as shown in the video)

Driving the face (verified client methods)

# Continuous facial parameters from the operator's face tracker
# (e.g. MediaPipe landmarks mapped to Furhat's parameter set)
furhat.request_face_params(params={...})

# Head orientation follows the operator's head
furhat.request_face_headpose(yaw=0.1, pitch=0.0, roll=0.0, relative=True)

In the video's live demo, the operator's video feed carries a face-detection box, and a second view shows the facial landmark mesh (eyes, brows) being tracked. Those landmarks are exactly what gets streamed as facial parameters.

How to build it, in order

The demo video

"Furhat Realtime API Demo | Gabriel Skantze" (Furhat Robotics, 12:07). This is the video all five workflows above come from. Watch it here:

Direct link: https://youtu.be/WtL7tHFHFJo

Download the starter files

Everything you need to run this guide. Click to open or download each file:

Verified method reference (package v0.1.3)

Troubleshooting

How this maps to your trust study

• Adaptive repair condition: 02_chatbot.py is the starting point. Instead of one fixed apology, the LLM writes a fresh repair message each time, informed by the conversation history.
• Scripted repair condition: a single furhat.request_speak_text(...) call with the same fixed apology text every time.
• Local models (Part 3): use this if the lab wants participant voice data to stay on-premise.
• Vision (Part 4): user tracking and attention control can log where participants look during the violation and repair, a useful behavioral measure.