The development of 3D imaging applications such as bin picking or object recognition can be very complex. The Ensenso range of stereo 3D cameras from IDS, with the latest SDK, has the necessary tools to meet these challenges. For example in multi-camera applications or in combination with robotics, complex calibrations of several coordinate systems are necessary before the data can be used effectively. Due to the strong system dependency, the application often has to be developed directly at the system level in order to generate usable data. In addition, higher resolution 3D stereo cameras, faster applications and embedded vision requirements all place high demands on the management of 3D data processing.
Basic data processing
Stereo 3D cameras such as the Ensenso series from IDS Imaging Development Systems GmbH use two cameras to acquire images from the same scene from different angles. Special matching algorithms compare the two images, search for corresponding points and visualize all point displacements in a Disparity Map. The quality of this matching process depends directly on the lighting of the scene and object surface textures. To improve the the perceived texture for objects with little natural texture, a high-intensity projector is used to project a random dot pattern onto the object’s surface. By moving the pattern across the surface, a varying texture is created. Acquiring multiple image pairs with these different textures results in improved resolution and reduced noise, but generates more data for processing. Sophisticated Semi-Global Matching (SGM) or Sequence Correlation (SC) algorithms can be used to process these image pairs.
Accelerated processing for embedded solutions
Ensenso models featuring Sony 5 Mpixel resolution sensors have recently been launched. Although this also increases the field of view by approximately 20% compared to the previous 1.3 MP sensors, the increase in resolution is accompanied by 4 x more data, with half the image acquisition rate, leading to potentially longer processing times to create a complete 3D image. However, the Sequence Correlation algorithm allows a complete 3D image calculation from 16 high-resolution 5 MP image pairs in about 2.5 seconds. This speed is sufficient for most applications. For applications with higher speed requirements, the semi-global matching algorthm provides sufficient accuracy using a single 5 MP image pair and a calculation time of only 1.1 seconds. Processing times can be reduced by utilising the processing power of a GPU. The latest version of the Software Development Kit for the Ensenso cameras features support for CUDA, NVIDIA’s parallel computing architecture. By optimising the algorithms for CUDA, processing speeds are about five times faster compared to CPU processing, depending on the GPU used and the parameterization of the corresponding algorithms. CUDA support also opens up opportunities for 3D applications within an embedded environment. For example the stereo calculations can access 256 CUDA cores on a platform such as the NVIDIA Jetson TX2 board, while subsequent image processing can be performed with ‘HALCON for Embedded Devices’ software running on an ARM CPU to provide a complete embedded solution.
Multi camera integration
Multi-camera applications provide different views and positions, and the coordinate systems of all of the cameras must be adjusted to each other or to fixed points in the real world and calibrated to a uniform object coordinate system. If cameras are to work with a robot and its movements must be coordinated with the camera data, a hand-eye calibration can also be performed. Specially developed calibration patterns also facilitate setup and adjustment in multi-camera systems. The Ensenso software library recognizes object coordinate systems of any size and the camera positions relative to each other using several calibration plates, some of which are covered by the cameras, and can then synchronize them with each other. The quality of inspection and measurement results in 3D applications can be significantly improved by also utilising the capabilities of 2D cameras. Where stereo cameras have difficulties identifying objects in border areas, 2D cameras perfectly assist by edge detection or colour recognition and can also allow the acquisition of additional information such as barcode content. It is particularly useful, therefore, to be able to integrate and calibrate 2D cameras into the same application.
Managing data for developers
A number of tools can be of great assistance for application developers. The availability of a ‘file camera’ that behaves like a real camera but uses stored images, allows application sequences to be simulated and run as often as necessary in order improve algorithms and processes, without the need to access the real system or recreate situations. Another useful tool is the ‘virtual camera’. Simulations can be performed in an offline environment, e.g. to evaluate data quality, completeness, resolution and noise of a scene with a particular camera model. For example, bin picking applications with different parts and their orientation in boxes can be easily simulated. An integrated scene editor including a random function can provide an unlimited number of variants for evaluation, equivalent, for example to unsorted part feeding in a real application. With virtual camera versions of all Ensenso models available, simulations can be run to evaluate the most suitable camera for the application.
