agent window: screen directions, pointing ends the turn
Deploy web Studio / deploy (push) Successful in 3m33s

Found while recording a demo with Qwen3.8 on the inference server:

- When the model wrote its answer and then called select_object, it was
  asked again with nothing left to say and rambled ("the user hasn't sent
  a message yet"), sometimes in Chinese. A step of page tools only
  (select_object, set_view) after a written answer now ends the turn; the
  system prompt says so, and to answer in the language of the latest message.
- The system prompt names the world directions of the user's screen right
  and up, and the 2:1 foreshortening, so "left of the hall" and shapes meant
  to be read on screen come out right (asked for a giant W of people, the
  model now draws it upright and readable).
- mock_llm.py "show me X" + check: answer and select_object in one step end
  the turn without another request, and the history stays valid.
This commit is contained in:
grabowski committed 2026-10-23 20:29:17 +07:00
1 parent 36f7a9f440
commit 84309a7a08
3 files changed
+34 -3

No files matched your search

+11
View File
@@ -81,6 +81,17 @@ def run(pg, objects, shot, check, can_set_key):
pg.wait_for_function("() => !document.querySelector('.ag-busy')", timeout=10000)
check(('"id":"%s"' % tid) in chats()[-1]["body"]["messages"][0]["content"], "the system prompt names the selection")
# an answer that ends by pointing at an object ends the turn there
n_chat = len(chats())
pg.locator("#agtext").fill("show me " + tid)
pg.locator("#agtext").press("Enter")
pg.wait_for_function("() => !document.querySelector('.ag-busy')", timeout=10000)
pg.wait_for_timeout(600)
check(len(chats()) == n_chat + 1 and ("id " + tid) in pg.locator("#right .st-sub").inner_text(),
"answer + select_object ends the turn (no extra request), the object is selected")
tail = pg.evaluate("() => __agent.A.msgs.slice(-2).map(m => m.role + ':' + (m.tool_call_id || (m.tool_calls || []).length))")
check(tail == ["assistant:1", "tool:call_s"], f"the history stays valid for the next request {tail}")
# Undo takes the agent's edit back
pg.locator("#undo").click()
pg.wait_for_timeout(700)
+8
View File
@@ -7,6 +7,8 @@ Wants the key `test-key`. Answers like a model that uses tools:
list_objects {type: tree} and add_object {type: tree, x: X, y: Y, name: "Agent tree"}
- tool results -> a short answer naming the new object's id in backticks
- a user message containing "slow" -> a long, slow answer (to test Stop)
- a user message "show me X" -> an answer and a select_object call for X in one step
(that ends the turn: no request follows)
- anything else -> a plain answer
Streams (SSE) when asked to. Every page origin may call it (CORS, also the
preflight). GET /_requests lists the requests seen (headers and bodies).
@@ -76,6 +78,12 @@ class H(BaseHTTPRequestHandler):
if last.get("role") == "user" and "slow" in last.get("content", ""):
deltas = [{"content": "word%d " % i} for i in range(80)]
return self.stream(body, deltas, finish, pause=0.1)
show = re.search(r"show me (\S+)", last.get("content", "")) if last.get("role") == "user" else None
if show:
deltas = [{"content": "Here is `%s`." % show.group(1)},
{"tool_calls": [{"index": 0, "id": "call_s", "type": "function",
"function": {"name": "select_object", "arguments": json.dumps({"id": show.group(1)})}}]}]
return self.stream(body, deltas, "tool_calls")
m = re.search(r"tree at (-?[\d.]+),\s*(-?[\d.]+)", last.get("content", "")) if last.get("role") == "user" else None
if m:
x, y = float(m.group(1)), float(m.group(2))