Users test Needle 3 demo and find tool selection inconsistent
3Sep 18 1:11 PM · 9d ago · 2 comments · 1 source · development 3 of 7
Testing showed the model handles direct commands like "turn all the lights on/off" reliably but fails on indirect requests. One tester reported "it's too cold" turned the thermostat down while "it's cold" dimmed lights, though confidence scores on these bad responses were low.
“"turn all the lights on/off" and "it's too dark in the bathroom" worked for me, but anything less direct didn't.”
"turn all the lights on/off" and "it's too dark in the bathroom" worked for me, but anything less direct didn't. "it's too cold" actually made it turn the thermostat down ("it's cold" made it... turn the lights down?)! Although the confidence on the bad responses was pretty low, so it might be worth adding a threshold to the demo.Or maybe it just…
thanks for these haha, you can actually edit the tools and/or their descriptions, the demo is just a "get started" preset. But still we do have room for reasoning improvement!