Why you do not use som image in workarena?

[leoozy]

[snehasquasher]

i had the same question!

[gasse]

Hi @leoozy @snehasquasher , in our latest experiments we did use screenshots with Set-of-Marks with a GPT4o-V agent, see the latest version of the paper here:
https://arxiv.org/abs/2403.07718

[leoozy]

Hello, does BrowserGym supports image-som?
I do not find it in the current obs.

   ```

obs = {
"chat_messages": copy.deepcopy(self.chat.messages),
"goal": goal_msg, # TODO: redundant with chat messages, to be removed?
"goal_image_urls": goal_image_urls, # TODO: redundant with chat messages, to be removed?
"open_pages_urls": [page.url for page in self.context.pages],
"active_page_index": np.asarray([self.context.pages.index(self.page)]),
"url": self.page.url,
"screenshot": extract_screenshot(self.page),
"dom_object": dom,
"axtree_object": axtree,
"extra_element_properties": extra_properties,
"focused_element_bid": focused_element_bid,
"last_action": self.last_action,
"last_action_error": self.last_action_error,
"elapsed_time": np.asarray([time.time() - self.start_time]),
}