From Language to Executable Actions

At Southern University of Science and Technology, I built the integration between an LLM agent and a JAKA Zu5 robotic arm. The agent translated natural-language requests into structured tool calls; ROS middleware dispatched the corresponding actions and returned execution feedback before the next step.

I designed the tool-calling interface and action middleware, then integrated them with MoveIt and a corrected robot model. Schema validation rejected malformed arguments, while joint limits and collision geometry constrained motion planning. Keeping execution sequential gave the agent a concrete result to reason from at each stage.

Closing the Loop on Hardware

The complete language-to-motion sequence was demonstrated in RViz/Gazebo and on the physical JAKA Zu5. Correcting link frames, collision meshes, and the gripper configuration helped carry the same control interface from simulation to hardware.

Vision-based object localization extended the workflow to pick-and-place interactions. Lighting and hand-eye calibration remained practical constraints, so execution feedback stayed central to the design: the next instruction depended on what the robot had actually completed.