Tasked with developing a conveyor tracking system with on-the-fly robot picking of bags of bread for an industrial bakery, the team at knew that machine vision was the only possible solution. KINE is a Finnish provider of custom turn-key robotics solutions, whose “key suppliers consider the preferred option for machine vision solutions,” notes Kimmo Salonen, chief technology officer at KINE. Helping KINE through their development and acquisition process was , a member of the one of the largest vendors of industrial components in northern Europe.
Prior to upgrading to a high-performance vision system, the picking process involved manual removal and depositing of bread packages. The challenge in designing a new system was that product sizes are irregular, and shapes differ from one type of bread to another, making it tough to determine their location and orientation on the conveyor. The transparent plastic packaging is also difficult to detect with optical sensors, due to low contrast and partial reflections. A robotic solution would ensure a faster, reliable, and cost-effective manner of product detection and processing (Image 1).
Image 1: The 3D conveyor tracking vision system offers on-the-fly robot picking
“We were already familiar with the capabilities of vision software, which has proven to be a versatile tool for this project, offering exactly the kind of integration and control we wanted,” says Salonen. As a result, KINE developed the vision system around this flowchart-based software and a 3D Time-of-Flight (ToF) camera as the sensor, selected as it eliminated the need for separate lighting, and reduced concerns about contrast and reflections.
Designed to function completely autonomously, the vision system sits directly atop the incoming product conveyor (Image 2). It centers on Matrox Design Assistant X software and the Basler 3D ToF camera, which affords real-time measurement of up to 30 fps and provides precise 3D data to the vision software. The camera is triggered from a photodetector by way of a PLC communicating with the vision program via a UDP; the PLC also synchronizes the robot by similar means. The Matrox Design Assistant X runtime environment and the associated vision program runs on an industrial panel PC. A Stäubli TS2-60 SCARA robot does the pick-and-place, communicating via TCP with Stäubli VALtrack software library.
Image 2: Mounted beside the robot, the ToF camera views the packaged bread and sends 3D data to vision software for processing and analysis.
Camera data analysis is implemented using Matrox Design Assistant X, which also produces the central interface to the vision system. The ToF camera records the surfaces as a 3D point cloud with over 300,000 XYZ coordinates; the software converts the point cloud into a depth map, which is analyzed using 2D vision tools to determine the grip points for the robot. The 2D tools provided by Matrox Design Assistant X work particularly well on depth-map data and allowed the team at KINE to effectively solve their application.
Sami Sinisalo, robot specialist at KINE, notes that, “there was a plan to use 3D information from the Basler ToF camera to determine the pick height for the robot, but we ultimately decided to create a recipe for each product with Matrox Design Assistant X.” Transparent plastic bags are not visible to the ToF camera (Image 3), and as the bread products within are arranged in random positions, the lack of a single, consistently flat surface makes it very difficult to determine the correct pick-height based on 3D depth-map information (Image 4). Recipes are used to configure select flowchart steps so that a flowchart can inspect different product variants. This allowed a single project to inspect the different types of breads. Recipes can be created and configured in Matrox Design Assistant X vision software at both design-time and runtime.
Image 3: The transparent plastic bag is not seen by the ToF camera.
Image 4: Rolls inside plastic bags are randomly arranged, making it difficult to determine correct pick height based on depth-map information.
KINE and their customer are very pleased with the results, which provides a lower error-rate than the prior manual handling and with a higher throughput of goods. KINE also saw a reduction in the overall costs related to machine vision by 75%, compared to traditional camera and lighting options. With the new vision system, the robot grip can handle between 25 and 30 packages per minute. The team at KINE sees many opportunities for future integration of Matrox Imaging products into their customed robot solutions, thanks in no small part to the success achieved with this 3D pick-and-place robot system.
Watch a video of this vision installation.
For more information, contact Matrox Imaging Media Relations.
