agent window: screen directions, pointing ends the turn
Deploy web Studio / deploy (push) Successful in 3m33s

Found while recording a demo with Qwen3.8 on the inference server:

- When the model wrote its answer and then called select_object, it was
  asked again with nothing left to say and rambled ("the user hasn't sent
  a message yet"), sometimes in Chinese. A step of page tools only
  (select_object, set_view) after a written answer now ends the turn; the
  system prompt says so, and to answer in the language of the latest message.
- The system prompt names the world directions of the user's screen right
  and up, and the 2:1 foreshortening, so "left of the hall" and shapes meant
  to be read on screen come out right (asked for a giant W of people, the
  model now draws it upright and readable).
- mock_llm.py "show me X" + check: answer and select_object in one step end
  the turn without another request, and the history stays valid.
This commit is contained in:
grabowski committed 2026-10-23 20:29:17 +07:00
1 parent 36f7a9f440
commit 84309a7a08
3 files changed
+34 -3

No files matched your search

+11
View File
@@ -81,6 +81,17 @@ def run(pg, objects, shot, check, can_set_key):
pg.wait_for_function("() => !document.querySelector('.ag-busy')", timeout=10000)
check(('"id":"%s"' % tid) in chats()[-1]["body"]["messages"][0]["content"], "the system prompt names the selection")
# an answer that ends by pointing at an object ends the turn there
n_chat = len(chats())
pg.locator("#agtext").fill("show me " + tid)
pg.locator("#agtext").press("Enter")
pg.wait_for_function("() => !document.querySelector('.ag-busy')", timeout=10000)
pg.wait_for_timeout(600)
check(len(chats()) == n_chat + 1 and ("id " + tid) in pg.locator("#right .st-sub").inner_text(),
"answer + select_object ends the turn (no extra request), the object is selected")
tail = pg.evaluate("() => __agent.A.msgs.slice(-2).map(m => m.role + ':' + (m.tool_call_id || (m.tool_calls || []).length))")
check(tail == ["assistant:1", "tool:call_s"], f"the history stays valid for the next request {tail}")
# Undo takes the agent's edit back
pg.locator("#undo").click()
pg.wait_for_timeout(700)