An important European robotics project called CHRIS (Cooperative Human Robot Interaction Systems FP7 215805) has received its final review last april. They have also created a very nice video that summarizes their work:
As the video show, the work includes the recognition of speech, gesture (pointing), actions, and objects. All within a context of cooperation and safety. But, I will not try to summarize their work. Just watch the video.