Heatmap motion analysis of autonomous robot

6 Comments

I have always wanted to ease my daily life and obtain vacuum robot. Many things were stopping for a while, but finally found a local company which provides demo units so could not resist taking one for a spin. What a good chance to run some computer vision processing algorithms to analyze it!

Seen few pictures where robot owners installed LED on top of their robot and took long exposure pictures capturing motion path in a single shot. I wanted to be more scientific and replicate this effect only without a LED and long exposure photography.

Preparation

As cleaning properties were reviewed by multiple owners I will skip this part. Let’s just say robot does what it was designed to do. I wanted to visualize motion path and coverage area. So placed couple of simple obstacles and added some dirt imitation in blue masking tape marked region (I thought robot will stay longer in this place but my guess was wrong).

img_8444_rCould not resist and mounted GoPro action camera to take awesome first person (FPV) video. Note: speed is increased 4 times.

robot1Installed Kurokesu C1 camera with 1.55m CS fisheye lens mounted above the room recorded all the action. Later this footage was used to run motion analysis.

img_8478_r

Results

Sketchy code I wrote over half an our uses Python and background subtraction functions from OpenCV, probably not even worth publishing as a functional program. You can clearly see persistent motion as a heat-map and and cleaned area. What a hypnotic view! (Full resolution youtube video will open if you click this animated gif)

blue1

Code highlights

Open video file and prepare for reading it.

cap = cv2.VideoCapture("robot_video.avi")

Initialize background subtraction function. Parameters might vary depending on your situation.

backsub = cv2.createBackgroundSubtractorMOG2(history=200, varThreshold=128, detectShadows=True)

Read and subtract background from each frame. Result image is all black. Motion areas are white.

ret, im = cap.read()   
fgmask = backsub.apply(im, None, 0.01)

Integrate all motion images into one frame. Kind of long exposure imitation.

arr = arr + imarr

After all frames are processed normalize result picture and run grayscale to heatmap gradient function. Some tricks were used to convert image into float type and normalize pixel values correctly before this step.

heat = cv2.applyColorMap(arr, cv2.COLORMAP_JET)

Save and display calculated heatmap picture

cv2.imwrite("heatmap.jpg", heat)
cv2.imshow("heatmap", heat)

Also added feature to save each n’th heatmap frame to decimate output clip and speed up processing time. After separate frames were produced video and animated gif’s were rendered.

Comments ( 6 )

  1. A Glimpse Into The Mind Of A Robot Vacuum Cleaner | HackerWorld
    […] [Saulius] decided to do it by videoing his robot with a fisheye lens from near the ceiling and then making a heatmap of the result. Not being satisfied with just a finished photo, he made a video showing the path taken as the room […]
  2. ReplyNickname ( required )
    Nice! Would be intresting to see how other cleaners performed in direct comparison. Thak you for sharing this!
    • Replysaulius
      Will publish few other later
  3. ReplyCarl Smith
    It could be interesting to modify your software to not quite totally subtract the background image, but leave just enough that we could see in the video what the robot is navigating around. Sort of like the new FLIR thermal cameras that have a normal optical camera that overlays a bit of a normal image over the thermal image so you can see what the thermal image maps onto in the real world.
    • Replysaulius
      Hi Carl, actually did this but for single frame. Extracted contours from original frame and overlaid heatmap picture. Sure it is more informative.
  4. » Robotic vacuum cleaners compared like never before
    […] computer vision algorithms to get live heat map. Result is well described in previous blog record heatmap video of one robot. This time I present you comparison between Vorwerk Kobold VR200, iRobot Roomba 980 and Neato […]

Leave a reply

Your email address will not be published.