User manual v1.2
Contents
1. 210 S P 1 Preview System Status Preview Processing Settings Cameras Trigger Time Sync Calibrate Cameras Review Calibration Network Settings Maintenance Help aa Display mode Red blue 29 6 fps Please note that this is only a preview without subpixel optimization which might not be displayed at the full frame rate Figure 11 Screenshot of configuration preview page Raw data The raw disparity data without color coding The pixel inten sity matches the integer component of the measured disparity Invalid disparities are displayed in light gray 8 3 Processing Settings The major processing parameters can be changed on the processing settings page which is shown in Figure The most relevant option is the operation mode which can be set to one of the following values Pass through In this mode the SP1 forwards the imagery of the selected cameras without modification This mode is intended for verifying that both cameras are functioning correctly Rectify In this mode the SP1 transmits the rectified images of both cameras This mode is intended for verifying the correctness of image rectification Stereo matching In this mode the SP1 performs stereo matching It trans mits the rectified image of the left camera and the left camera disparity map 15 8 3 Processing Settings 8 CONFIGURATION SP1 Chromium SP1 x WR 2 lt 5 192 168 10 10 LO S P 1 Stereo Proc2. 5 192 168 10 10 S P 1 Time Synchronization System Status NTP Synchronization Synchronize clock to NTP server Processing Settings NTP Server 192 168 10 5 Change ee Calibrate Cameras Pulse Per Second PPS l Review Calibration Reset clock on PPS pulse u Sync Network Settings change LE Ground PPS pulse must have a voltage level of 3 3 V Activating PPS will disable trigger 1 A Trigger 0 Trigger 1 Figure 15 Screenshot of configuration page for time synchronization 8 7 Camera Calibration 8 7 1 Calibration Board The calibrate cameras page which is shown in Figure 16 enables the calibration of the stereo camera pair You require a calibration board which is a flat panel with a visible calibration pattern on one side The pattern that is used by the SP1 consists of an asymmetric grid of black circles on a white background as shown in Figure The pattern can be downloaded directly from the calibration page Simply select the desired paper size in the calibration board drop down list and click the download link Even if you already have a calibration board make sure to always select the correct board size before starting the calibration process Otherwise the calibration results cannot be used for 3D reconstruction with a correct metric scale see Section 7 2 Should you require a calibration board with a custom size then you can select custom from the calibration board drop down list This allow
3. However you have to ensure that the network is capable of handling the high bandwidth data that is transmitted by the device The network must support data rates of at least 25 MB s The cameras are connected to the SP1 s USB port As the SP1 only features one USB port a USB hub is mandatory for making this connection Please note that the USB hub usually has to be powered externally to meet the power consumption of the cameras 6 2 Supported Cameras The SP1 supports the following camera models e Point Grey Blackfly model BFLY U3 0352M CS e Point Grey Blackfly model BFLY U3 1352M CS e Point Grey Chameleon3 model CM3 U3 1352M CS e Point Grey Chameleon3 model CM3 U3 13Y3M CS e Point Grey Flea3 model FL3 U3 13E4M C e Point Grey Flea3 model FL3 U3 13Y3M C e Point Grey Grasshopper3 model GS3 U3 1455M C e Point Grey Grasshopper3 model GS3 U3 1555M C 6 3 Camera Alignment 6 HARDWARE SETUP Figure 6 Example for standard epipolar geometry e IDS uEye ML model UI 3240ML M GL e IDS uEye CP model UI 3140CP M GL Rev 2 e IDS uEye CP model UI 3240CP M GL All cameras must have a gray scale sensor 6 3 Camera Alignment Both cameras have to be mounted on a plane with a displacement that is perpendicular to the cameras optical axes Furthermore both cameras must be equipped with lenses that have an identical focal length This arrangement is known as the standard epipolar geometry An example for such a camera mounti
4. The cameras page that is shown in Figure allows for the selection of a desired camera pair and the adjustment of their respective parameters All detected cameras are listed in the camera selection list A source is listed for each camera which identifies the utilized camera driver The cameras list also includes the two virtual camera pairs that are mentioned in Section 6 4 which provide an example stereo sequence or facilitate the reception of input images through ethernet To choose a particular camera you have to tick either the use as left or use as right radio button for selecting the camera as the left or right camera of the stereo pair Please note that cameras of different sources cannot be used together 8 4 2 Camera Settings After selecting a camera pair camera specific settings can be adjusted in the left camera settings and right camera settings area The available settings de pend on your camera model Please refer to your camera documentation for details Some of the more advanced settings of your camera might not be available In this case please use the software provided by the camera manu facturer to adjust the desired settings and safe the new camera configuration to the cameras internal memory 17 8 4 Cameras 8 CONFIGURATION SP1 Chromium SP1 x un lt 3 192 168 10 10 w O S P 1 Cameras Camera Selection Source ID Use as left Use as right USB uEye 410288
5. 2 Disparity Maps 7 PROCESSING RESULTS Figure 9 Example for a left camera image and corresponding disparity map 7 2 Disparity Maps The stereo matching results are delivered in the form of a disparity map from the perspective of the left camera The disparity map assigns each pixel in the left camera image to a corresponding pixel in the right camera image Because both images were previously rectified to match an ideal standard epipolar geometry corresponding pixels should only differ in their horizontal coordinates The disparity map thus only encodes a horizontal coordinate difference An example for a left camera image and the corresponding disparity map are shown in Figures a and Ob Here the disparity map has been color coded with blue hues reflecting small disparities and red hues reflecting large dispar ities The disparity is proportional to the inverse depth of the corresponding scene point It is thus possible to transform the disparity map into a set of 3D points This can be done at a correct metric scale if the cameras have been calibrated properly 11 7 3 Timestamps and Sequence Numbers 7 PROCESSING RESULTS The transformation of a disparity map to a set of 3D points requires knowl edge of the disparity to depth mapping matrix Q which is computed during camera calibration and transmitted by the SP1 along with each disparity map The 3D location z y 2 of a point with image coordinates u v and dis parit
6. Figure 4 Interfaces on a front and b rear housing side Power jack 5 Physical Interfaces Figures Ha and 4b show the interfaces on the SP1 s front and backside The power jack is located on the backside and it needs to be connected to the supplied power adapter or an equivalent model When using an alternative power supply please make sure that the voltage is set to 5 V DC Higher voltages might damage the device The front side features the following interfaces Power LED Indicates that the device is powered up and running Busy LED Indicates that the device is currently processing image data Trigger port Port for providing two camera trigger signals It is possible to disable one trigger signal and use the dedicated pin as input for a synchronization signal Please see Sections 6 5 and 6 6 Ethernet port Port for connecting to a client computer It is used for de livering processing results and for providing access to the configuration interface USB port Port for connecting the desired USB cameras through a USB hub 6 HARDWARE SETUP USB cameras Computer Ethernet Figure 5 Example setup for cameras SP1 and client computer 6 Hardware Setup 6 1 Connecting the SP1 Figure 5 shows a basic system setup for stereo vision A client computer that receives the processing results is connected to the SP1 s ethernet port Alternatively it is possible to connect the SP1 to a switched network
7. images The currently displayed images can be written to files by pressing the space key When pressing the enter key all received images are written to files up until enter is re pressed The created image files are located in the current working directory SpCom can be controlled through the command line options listed in Ta ble Most commonly SpCom is used to display the color coded disparity map that is transmitted by SP1 If using UDP as underlying network protocol and the default port number 7681 this can be achieved by executing SpCom with the following command line gt spcom c 11 Support If you require support or if you have other inquiries that are related to this product please contact Dr Konstantin Schauwecker Nerian Vision Technologies Gotenstr 9 70771 Leinfelden Echterdingen Germany Phone 49 711 2195 9414 E mail service nerian com Website http nerian com 32 11 SUPPORT Table 1 Available command line options for SpCom c s directory n d f number w directory h hostname p port H hostname P port Color code the disparity map Send images from the given directory If not specified only image reception is active Non graphical mode No display window will be opened Disable image reception Use this option if you want to trans mit images only Limits the frame rate to the given value This option can be used for controlling the frame rate when transmitting i
8. 05 44 46 daemon info stereod Fps 30 32 05 44 48 daemon info stereod Fps 30 00 05 44 50 daemon info stereod Fps 30 00 05 44 52 daemon info stereod Fps 30 01 05 44 54 daemon info stereod Fps 30 00 05 44 57 daemon info stereod Fps 30 01 05 44 59 daemon info stereod Fps 30 01 05 45 01 daemon info stereod Fps 29 99 05 45 03 daemon info stereod Fps 29 98 05 45 05 daemon info stereod Fps 30 01 05 45 07 daemon info stereod Fps 29 99 05 45 09 daemon info stereod Fps 30 06 05 45 11 daemon info stereod Fps 29 99 05 45 13 daemon info stereod Fps 29 75 05 45 15 daemon info stereod Fps 29 90 Figure 10 Screenshot of configuration status page between the reception of the synchronization signal and the time of capturing the current image pair 8 Configuration The SP1 is configured through a web interface which can be reached by en tering the IP address of the SP1 in your browser The default address is Please configure the IP address and subnet mask of your computer appropriately such that this interface can be reached If the SP1 has just been plugged in it will take several seconds before the web interface is accessible For using the web interface you require a browser with support for HTML 5 Please use a recent version of one of the major browsers such as Internet Explorer Firefox Chrome or Safari 8 1 System Status The first page that you see when opening the web interface is the system status
9. 4 84 add 9 4 3D Reconstruction 10 SpCom Sample Application 11 Support 12 Warranty Information 13 Open Source Information 28 28 29 30 31 31 32 34 34 3 GENERAL SPECIFICATIONS 1 Functionality Overview The SP1 stereo vision system is a stand alone processing system for performing stereo matching in real time It connects to two industrial USB cameras that provide input image data The SP1 correlates the images of both cameras and produces a disparity map which is transmitted through gigabit ethernet The disparity map describes a mapping of image points from the left camera image to corresponding image points in the right camera image With this information it is possible to reconstruct the 3D location of the corresponding scene points 2 Included Parts When ordering a new SP1 from Nerian Vision Technologies the package should include the following parts e SP1 stereo vision system e 5 V DC power adapter e 4 stickable rubber pads not included for surface mountable version e Manual If any of the listed parts is missing in your delivery please contact us 3 General Specifications 3 1 Hardware Details Power consumption lt 4 W Power supply 5 V DC Dimensions 105 x 76 x 36 mm Weight 0 25 kg I O USB 2 0 host gigabit ethernet 2x trigger out Max USB power 500 mA 3 2 Stereo Matching Stereo algorithm Semi Global Matching SGM Disparity range 112 pixels Processing rate 30 Hz S
10. 7663122271650 02 2 43R69RIA209KA2E 02 0 q 1 Bos907215314301150 02 0 3 5290341512990210e 12 Figure 19 Screenshot of configuration page for reviewing camera calibration Figure 20 Example for evaluating vertical image coordinates 25 8 8 Reviewing Calibration Results 8 CONFIGURATION M and Mz camera matrices for the left and right camera D and Ds distortion coefficients for the left and right camera R and Ra rotation matrix for the rotation between the original and rectified camera images P and P3 projection matrices in the new rectified coordinate systems Q the disparity to depth mapping matrix See Section 7 2 for its use T translation vector between the coordinate systems of both cameras R rotation matrix between the coordinate systems of the left and right cam era The camera matrices M and M are structured as follows fe 0 Es M 0 fy Cy 1 0 0 1 where f and f are the lenses focal lengths in horizontal and vertical direction measured in pixels and c and c are the image coordinates of the projection center The distortion coefficient vectors D and D have the following structure Di ki ko pi po ks 2 where k k2 and kz are radial distortion coefficients and p and p are tangential distortion coefficients You can download all calibration information as a machine readable YAML file by clicking the download link at the bottom of the calib
11. 9136 USB uEye 4102889137 Network network_left Network network_right Example example_left Example example_right ce Left Camera Settings Pixel binning 2x2 Pixel clock E 86 MHz Frame rate E 63 69 fps AOI Y position E 16 Gain mode Auto Exposure shutter mode auto Frame rate mode Manual Black level mode Auto vr Auto setpoint 100 LI Auto skip frames gd Figure 13 Screenshot of configuration page for camera settings By pressing the reset camera defaults button you can reset the camera s settings to the default configuration This is usually the configuration that has been written to the camera s internal memory though the manufacturer software If the reset button is pressed all configuration changes that have not yet been confirmed through the change button are reverted The SP1 offers the functionality to synchronize parameters that are other wise automatically adjusted by each camera For example each camera might control its shutter time through an auto shutter algorithm The SP1 can syn chronize such auto adjusted parameters by setting the parameters of the left camera to always match the respective parameters of the right camera If this functionality is desired the mode for the particular feature has to be set to synchronize to left camera in the right camera settings and to auto in the left camera settings In this case the SP1 will query the settings of the left camera and adj
12. P1 xo D 192 168 10 10 m vo S P 1 Maintenance BER Reboot now Firmware Update Current version 1 1 0 WARNING Do not unplug device reload this page or re click update button while performing firmware updates Calibrate Cameras This might lead to a currupted firmware which can only be corrected by our service personnel Review Calibration Choose File No file chosen Update Network Settings Maintenance A Figure 22 Screenshot of configuration maintenance page The maintenance page further allows you to perform firmware updates Use this functionality only for firmware files that have officially been released by Nerian Vision Technologies To perform a firmware update select the desired firmware file and press the update button The update process will take several seconds Do not unplug the device reload the maintenance page or re click the update button while performing firmware updates Otherwise this might lead to a corrupted firmware state 9 API Usage Information 9 1 General Information The cross platform libvisiontransfer C API is available for interfacing cus tom software with the SP1 For windows a binary version of the library is available that can be used with Microsoft Visual Studio 2013 and 2015 For Linux and Mac OSX please compile the library from the available source code package The API can be downloaded from the product websitd The libvisiontransfer API provides functionally for recei
13. SP1 Stereo Vision System User Manual v1 2 November 23 2015 nerian VISION TECHNOLOGIES Dr Konstantin Schauwecker Nerian Vision Technologies Gotenstr 9 70771 Leinfelden Echterdingen Germany Email service nerian com http nerian com Contents 1 Functionality Overview 2 Included Parts 3 General Specifications 3 1 Hardware Details 8 4 4 4 a a ee een 3 2 Stereo Malchmel sem msgs a QUE a es SO Rw Re 3 3 Image Rectification lt 2 4 0 5 gas kee we BRS Se eS 4 Mechanical Specifications 4 1 Dimensions 4 2 Mounting 22 25 24 5 Physical Interfaces 6 Hardware Setup 6 1 Connecting the SPl ZITTE 6 3 Camera Alignment ea uses 6 6 Time Synchronization Signal een ne ee Gayton saz 8 Configuration 8 1 System Status Sank A ea a ee ee 8 3 Processing Settings 8 4 Cameras 8 7 8 4 1 Camera Selection 8 4 2 Camera Settings er ERTEILEN samera Calibration 8 7 1 Calibration Board 8 7 2 Recording Calibration Frames 2 2 2 2 2 8 7 3 Performing Calibration 10 10 11 12 8 8 Reviewing Calibration Results 222222 20 8 9 Network Settings 2 2 2 2 24 bo 224 ar as nu 8 10 Maintenance 9 API Usage Information 9 1 General Information 0 0 0 0002 eee 9 2 ImageTransfer Example 22 4 9 3 syneTransfer Example 2 2 84 2a 4 6
14. al for the respective other camera As an alternative the SP1 can produce up to two trigger signals The signals are provided through the trigger port which can receive a standard 3 5 mm phone connector The pin assignment for this connector is shown in Figure The peak voltage of both trigger signals is at 3 3 V The polarity of the trigger signals is active high Cameras are allowed to trigger on either a rising 6 6 Time Synchronization Signal 7 PROCESSING RESULTS or a falling edge The pulse width and frequency can be adjusted in the trigger configuration see Section 8 5 To protect the trigger port from short circuits each signal line is connected through a 220 Q series resistor 6 6 Time Synchronization Signal As indicated in Figure 7 the pin dedicated to trigger signal 1 can alternatively be configured as input for a time synchronization signal see Section 8 6 In this case the internal clock is set to 0 whenever a rising signal edge is received The signal must have a voltage level of 3 3 V Please avoid higher voltages as they might damage the device Clock synchronization is useful when interpreting the timestamps that are embedded in the transmitted processing results see Section 7 3 The syn chronization input can be connected to the pulse per second PPS output of a GPS receiver or a precision oscillator in which case the clock is reset once per second This allows for the reconstruction of high precision timesta
15. asler NVIDIA Corporation OpenCV Foundation Pablo Aguilar Peter Ross Phas eSpace Inc Philipp Wagner Pixar Animation Studios Reimar D ffinger Rob Hess Ryan Martell Sam Leffler Samuel Pitoiset Silicon Graphics Inc Simon Perreault Smartjog S A S Splitted Desktop Systems Takuji Nishimura The Khronos Group Inc The Regents of the University of California the Sphinx team The University of British Columbia the Wine project Thomas G Lane University of Augsburg University of Delaware Weta Dig ital Ltd William Lucas Willow Garage Inc Xavier Delacour Yossi Rubner If you believe that your name should be included in this list then please let us know 36 Revision History 13 OPEN SOURCE INFORMATION Revision History Revision Date Author s Description v1 2 Nov 23 2015 KS Updated for firmware 1 1 0 and libvisiontransfer 2 0 0 time syn chronization network image in put timestamps sequence num bers various smaller changes v1 1 Sept 21 2015 KS Speckle filter corrections to open source information v1 0 Aug 28 2015 KS List of compatible cameras in formation on IDS cameras soft ware download link extended open source information general im provements v0 1 July 30 2015 KS Initial revision Gef rdert durch R Bundesministerium AR f r Wirtschaft und Energie aufgrund eines Beschlusses des Deutschen Bundestages Existenzgr ndungen aus der Wissenschaft 37 exist AESF EH Europ
16. ble to synchronize the SP1 s internal clock to an external signal or a time server This directly affects the produced time stamps When synchronized to a time server time stamps are measured in microseconds since 1 January 1970 00 00 00 UTC If no synchro nization is performed the internal clock is set to 0 at the time the SP1 is powered up If synchronizing to an external signal the clock is set to 0 every time the synchronization signal is asserted Please note that synchronizing to a PPS signal as explained in Section 6 6 also produces negative timestamps This happens whenever a synchronization signal is received while the SP1 is still processing an image pair that has been captured earlier The negative timestamp is then the time difference 12 8 CONFIGURATION SP1 Chromium SP1 x u a 5 192 168 10 10 sta vo S P 1 System Status Information System Status Preview Processing status running Processing Settings SoC temperature 57 8 C Cameras Trigger Time Sync System Logs Calibrate Cameras 05 44 28 daemon info stereod Fps 30 00 05 44 30 daemon info stereod Fps 30 01 Review Calibration 05 44 32 daemon info stereod Fps 30 00 05 44 34 daemon info stereod Fps 29 99 Network Settings 44 36 Hace LATO stereod Fps 30 04 44 38 daemon info stereod Fps 30 00 Maintenance 05 44 40 daemon info stereod Fs 30 00 Help 05 44 42 daemon info stereod Fps 30 04 05 44 44 daemon info stereod Fps 30 00
17. class creates one or more threads that handle all network communication Detailed information on the usage of each class can be found in the available API documentation 9 2 ImageTransfer Example An example for using the class ImageTransfer to receive processing results over the network and writing them to image files is shown below This source code file is part of the API source code package Please refer to the API doc umentation for further information on using ImageTransfer include lt visiontransfer imagetransfer h gt include lt visiontransfer imagepair h gt include lt iostream gt include lt stdio h gt ifdef MSC VER Visual studio does not come with snprintf define snprintf _snprintf_s endif int main Create an image transfer object that receives data from the SP1 on the default port using UDP ImageTransfer imageTransfer ImageTransfer UDP NULL NULL NULL 7681 Receive 100 images for int i 0 i lt 100 i std cout lt lt Receiving image lt lt i lt lt std endl 29 9 3 AsyncTransfer Example 9 API USAGE INFORMATION Receive image ImagePair imagePair while imageTransfer receivelmagePair imagePair true Keep on trying until reception is successful Write both images one after another for int imageNumber 0 imageNumber lt 1 imageNumber Create PGM file char fileName 100 snprintf fileName sizeof fileName image 03d d pgm i i
18. data to any other host The transfer is continued as soon as the browser window is closed the user presses the pause button below the preview area or if the user navigates to a different page Only one open instance of the preview page or any other page that is streaming video data to the browser is allowed at a time If attempted to open more than once only one instance will receive data The preview that is displayed in the browser does not reflect the full qual ity of the computed disparity map In particular sub pixel accuracy is not available and your browser might not be able to display the disparity map at the full camera frame rate To receive a full quality preview please use the SpCom sample application which is described in Section Different color coding schemes can be selected through the drop down list below the preview area A color scale is shown to the right which provides information on the mapping between colors and disparity values The possible color schemes are Red blue A gradient from red to blue with red hues corresponding to high disparities and blue hues corresponding to low disparities Invalid disparities are depicted in black Rainbow A rainbow color scheme with low wavelengths corresponding to high disparities and high wavelengths corresponding to low disparities Invalid disparities are depicted in grey 14 8 3 Processing Settings 8 CONFIGURATION SP1 Chromium lt D 192 168 10 10
19. ehrson Google Inc Greg Ward Larson Group 42 Inc Guy Eric Schalnat Harlan Stenn Image Power Inc Indus trial Light amp Magic Institute Of Software Chinese Academy Of Science Intel Corporation Ivan Kalvachev James Almer Javier S nchez P rez Jean loup Gailly Joris Van Damme Joseph O Rourke Jostein Austvik Jacobsen Justin Ruggles Ken Turkowski Kevlin Hen ney GIPSA French Labs LISTIC Lab Liu Liu Maarten Lankhorst Makoto Matsumoto Mans Rullgard Marius Muja Mark Adler Martin Burnicki Martin Storsjo Michael David Adams Michael Niedermayer Microsoft Corporation Multicoreware Inc Nicolas George 18 OPEN SOURCE INFORMATION Table 3 Open source components Name Version License s busybox 1 22 1 GNU GPL 2 0 dropbear 2014 63 MIT License BSD License OpenSSH License glib 2 40 GNU LGPL 2 0 libdc1394 2 2 3 patched GNU LGPL 2 1 libffi 3 1 MIT License libusb 1 0 18 GNU LGPL 2 1 libwebsockets 1 3 GNU LGPL 2 1 libxml2 2 9 1 MIT License linux 3 18 0 xilinx GNU GPL 2 0 monit 5 13 GNU GPL 3 0 ntp 4 2 8 BSD License MIT License opencv 2 4 9 BSD License libpng License JasPer License 2 0 openssl 1 0 2c OpenSSL License PHP 5 5 16 PHP License Sourcery CodeBench 2014 11 30 GNU GPL 2 0 Lite for Xilinx GNU GPL 3 0 Cortex A9 GNU Linux u boot zlib xilinx v2015 2 1 2 8 35 Mentor Graphics License BSD Various GNU GPL 2 0 Zlib License 18 OPEN SOURCE INFORMATION Nils H
20. equency e If an automatic adjustment of gain exposure shutter and black level is desired then please apply the following settings For the left camera set the mode of gain exposure shutter and black level to auto For the right camera set the mode of gain exposure shutter and black level to synchronize to left camera Leave auto skip frames at 4 the default in order to minimize dropped frames during adjustments 8 5 Trigger The trigger page that is shown in Figure 14 allows for a configuration of the external trigger settings As described in Section 6 5 the SP1 features a trigger port that provides access to up to two trigger signals The two trigger signals trigger 0 and trigger 1 can be enabled or disabled by selecting the respective check boxes 19 8 6 Time Synchronization 8 CONFIGURATION SP1 Chromium SP1 x UR A SP1 5 192 168 10 10 Trigger Settings Trigger 0 Enabled Frequency 300 Hz Pulse width 50 ms Trigger 1 Enabled disables PPS synchronization Frequency Offset Pulse width Change 30 0 Hz 0 0 ms 5 0 ms m Trigger O Trigger 1 u Sync I Ground Figure 14 Screenshot of configuration page for trigger settings For trigger 0 it is possible to select a frequency between 6 and 50 Hz and an arbitrary pulse width in milliseconds The polarity of the generated trigger signal is active high The signal t
21. essor Settings Operation Mode Pass through i i Rectify Processing Settings ESB creme Algorithm Settings Stereo Matching Penalty for disparity changes P1 6 Penalty for disparity discontinuities P2 22 Post Processing Mask border pixels Consisteny check Consistency check sensitivity 13 Uniqueness check Uniqueness check sensitivity E 65 Gap interpolation Noise reduction Speckle filter Change Reset Defaults Figure 12 Screenshot of configuration page for processing settings If the operation mode is set to stereo matching then the behavior of the image processing algorithms can be controlled through the algorithm settings These settings include the following stereo matching parameters Penalty for disparity changes P1 A penalty that is applied to gradu ally changing disparities A large value causes gradual disparity changes to occur less frequently while a small value causes gradual changes to occur more frequently This value has to be smaller than P2 Penalty for disparity discontinuities P2 A penalty that is applied to abruptly changing disparities A large value causes disparity discontinu ities to occur less frequently while a small value causes discontinuities to occur more frequently This value has to be greater than P1 The SP1 implements several methods for post processing the computed disparity map Each post processing method can be activated or deactivated individual
22. etected calibration board is shown in Figure 18 If the calibration board was not detected correctly or if you are unhappy with the quality of a calibration frame then you can delete it by clicking on the x symbol 23 8 8 Reviewing Calibration Results 8 CONFIGURATION 8 7 3 Performing Calibration Once you have recorded a sufficient number of calibration frames you can initiate the calibration process by pressing the calibrate button in the control section The time required for camera calibration depends on the number of calibration frames that you have recorded Calibration will usually take several minutes to complete If calibration is successful then you are immediately redirected to the review calibration page Calibration will fail if the computed vertical or horizontal pixel displace ment exceeds the allowed range of 31 31 pixels for any image point The most common causes for calibration failures are e Insufficient number of calibration frames Improperly aligned cameras See Section Lenses with strong geometric distortions Lenses with unequal focal lengths e Improper camera synchronization e Frames with calibration board misdetections Should calibration fail then please resolve the cause of error and repeat the calibration process If the cause of error is one or more erroneous calibration frames then you can delete those frame and re press the calibrate button Likewise in case of too few calibration f
23. ischer Sozialfonds EUROP ISCHE UNION f r Deutschland
24. ly The available methods are Mask border pixels If enabled this option marks all disparities that are close to the border of the visible image area as invalid as they have a high uncertanty This also includes all pixels for which no actual image data is available due to the image rectification see Section 7 1 16 8 4 Cameras 8 CONFIGURATION Consistency check If enabled stereo matching is performed in both match ing directions left to right and right to left Pixels for which there is no consistent disparity are marked as invalid The sensitivity of the consis tency check can be controlled through the consistency check sensitivity slider Uniqueness check If enabled pixels are marked as invalid if there is no suf ficiently unique solution i e the cost function does not have a global minimum that is significantly lower than all other local minima The sensitivity of the uniqueness check can be controlled through the unique ness check sensitivity slider Gap interpolation If enabled then small patches of invalid disparities which are caused by either the consistency or the uniqueness check are filled through interpolation Noise reduction If enabled an image filter is applied to the disparity map which reduces noise and removes outliers Speckle filter Marks small isolated patches of similar disparity as invalid Such speckles are often the result of erroneous matches 8 4 Cameras 8 4 1 Camera Selection
25. mageNumber imagePair writePgmFile imageNumber fileName return O 9 3 AsyncTransfer Example An example for using the class AsyncTransfer to receive processing results over the network and writing them to image files is shown below This source code file is part of the API source code package Please refer to the API doc umentation for further information on using AsyncTransfer include lt visiontransfer asynctransfer h gt include lt visiontransfer imagepair h gt include lt iostream gt include lt stdio h gt ifdef MSC VER Visual studio does not come with snprintf define snprintf _snprintf_s endif int main Create an async transfer object that receives data from the SP1 on the default port using UDP AsyncTransfer asyncTransfer ImageTransfer UDP NULL NULL NULL 7681 Receive 100 images for int i 0 i lt 100 i std cout lt lt Receiving_image_ lt lt i lt lt std endl Alternatingly receive the first and second image ImagePair imagePair while asyncTransfer collectReceivedImagePair imagePair 0 1 xtimeoutx 30 9 4 3D Reconstruction 10 SPCOM SAMPLE APPLICATION Keep on trying until reception is successful Write both images one after another for int imageNumber 0 imageNumber lt 1 imageNumber Create PGM file char fileName 100 snprintf fileName sizeof fileName image 03d_ d pgm i imageNumber imagePair writePgmFile i
26. mageNumber fileName return O 9 4 3D Reconstruction As described in Section 7 2 the disparity map can be transformed into a set of 3D points This requires knowledge of the disparity to depth mapping matrix Q see Section 7 2 which is transmitted by the SP1 along with each disparity map An optimized implementation of the required transformation which uses SSE or AVX instructions is provided by the API through the class Recon struct3D This class converts a disparity map to a map of 3D point coordi nates Please see the API documentation for details on using this class 10 SpCom Sample Application The downloadable source code or binary version of the libvisiontransfer API also include the SpCom sample application which is shown in Figure 23 When compiling this application yourself please make sure that you have the libraries OpenCV and libSDL installed SpCom provides the following features e Receive and display images and disparity maps from SP1 e Perform color coding of disparity maps e Write received data to image files e Transmit image pairs to SP1 e Simulate SP1 by transmitting images and disparity maps SpCom should be run from a terminal in order to supply command line arguments and to view the printed status messages If the graphical mode 31 11 SUPPORT Figure 23 Screenshot of SpCom sample application is not disabled then SpCom opens up a window for displaying the received
27. mages Immediately start writing all received images to the given directory Use the given remote hostname for communication Only required for TCP communication and for transmitting im ages through UDP By default this parameter is set to 192 168 10 10 Use the given remote port number for communication Only required for TCP communication and for transmitting images through UDP By default this parameter is set to 7681 Use the given local hostname as local interface address If not specified the default interface will be used Use the given local port number for communication Only necessary for receiving images through UDP By default this parameter is set to 7681 Use TCP instead of UDP as the underlying network protocol Output the disparity map right image as 16 bit image with out bit depth conversion Only display image 0 left or 1 right 33 18 OPEN SOURCE INFORMATION 12 Warranty Information The device is provided with a 2 year warranty according to German federal law BGB Warranty is lost if e the housing is opened by others than official Nerian Vision Technologies service staff e the firmware is modified or replaced except for official firmware updates In case of warranty please contact our support staff 13 Open Source Information The SP1 firmware contains code from the open source libraries and applications listed in Table B Source code for these software components and the wording of the re
28. mps on the computer receiving the SP1 s processing results As an alternative to synchronizing to an external signal the SP1 can also perform a clock synchronization through the Network Time Protocol NTP as described in Section 8 6 7 Processing Results 7 1 Rectified Images Even when carefully aligning both cameras you are unlikely to receive images that match the expected result form an ideal standard epipolar geometry The images are affected by various distortions that result from errors in the cameras optics and mounting Therefore the first processing step that is performed is an image undistortion operation which is known as image rectification Image rectification requires precise knowledge of the cameras projective parameters These can be determined through camera calibration Please refer to Section 8 7 for a detailed explanation of the camera calibration procedure Figure 8a shows an example camera image where the camera was pointed towards a calibration board The edges of the board appear slightly bent due to radial distortions caused by the camera s optics Figure 8b shows the same image after image rectification This time all edges of the calibration board are perfectly straight When performing stereo matching the SP1 additionally outputs the rec tified left camera image This allows for a mapping of features in the visible image to structures in the determined scene depth and vice versa 10 7
29. ng is shown in Figure 6 The distance between both cameras is referred to as baseline distance Us ing a large baseline distances improves the depth resolution at high distances A small baseline distances on the other hand allows for the observation of very close objects The baseline distance should be adjusted in conjunction with the lenses focal length An online tool for computing desirable combina tions of baseline distance and focal length can be found on the Nerian Vision Technologies websitd 6 4 Other Image Sources The SP1 can alternatively process image data that does not originate from real cameras To allow for an easy evaluation each SP1 ships with an example stereo sequence on its internal memory This example sequence appears as two virtual cameras that can be selected during camera configuration see Section B 4 If selected the example sequence is replayed in an infinite loop Due to speed limitations of the internal memory the example sequence is not replayed at the full frame rate during the first loop iteration Another set of virtual cameras provide the ability to receive image data over the ethernet port In this case a computer transmits a set of image pairs to the SP1 and then receives the processing results over the same network http nerian com products sp1 stereo vision calculator 8 6 5 External Trigger 6 HARDWARE SETUP Trigger O Trigger 1 E Sync L Ground Figure 7 Pin as
30. ou can specify an IP address subnet mask and gateway address When changing the IP settings please make sure that your computer is in the same subnet or that there exists a gateway router through which data can be transferred between both subnets Otherwise you will not be able to access the SP1 In the communication settings section you can choose the underlying net work protocol that shall be used for delivering the computation results to the client computer The possible options are TCP and UDP Due to the high bandwidth real time data we recommend using UDP unless the input images are transferred through ethernet as described in Section If TCP is selected the SP1 opens up the server port 7681 and waits for a client computer to connect before transmitting data Because UDP is a connection less protocol data transmission starts instantly if UDP is selected In UDP mode you thus have to specify the IP address and port number of the client computer It is possible to enter a multicast address here if you want the data to be received by multiple hosts or processes 8 10 Maintenance The maintenance page which is shown in Figure allows you to trigger a reboot of the SP1 This can be done by pressing the reboot now button Perform this step if the SP1 has stopped functioning correctly When a reboot is triggered it will take several seconds before the device becomes be reachable again 27 9 API USAGE INFORMATION SP1 Chromium S
31. page that is shown in Figure On this page you can find the following information Processing status Indicates whether the image processing sub system has been started If this is not the case then there might be a problem 13 8 2 Preview 8 CONFIGURATION accessing the cameras or another system error might have occurred Please consult the system logs in this case The image processing sub system will be started immediately once the cause of error has been removed such as connecting the cameras SOC temperature The temperature of the central System on Chip SoC that performs all processing tasks The maximum allowed temperature is at 70 C A green orange red color coding is applied to signal good alarming and critical temperatures System logs List of system log messages sorted by time In regular operation you will find information on the current system performance In case of errors the system logs contain corresponding error messages 8 2 Preview The preview page which is shown in Figure provides a live preview of the currently computed disparity map Please make sure that your network con nection supports the high bandwidth that is required for streaming video data see Section 6 1 For using the preview page you require a direct connection to the SP1 An in between proxy server or a router that performs network address translation NAT cannot be used When opening the preview page the SP1 stops transferring image
32. press the capture single frame button in the control section Repeat this process several times while moving either the camera or the calibration board The calibration board must be recorded at several different positions and orientations You should also vary the distance of the board to the cameras and make sure that you cover most of the cameras field of view When recording the calibration frames it is important that both cameras are synchronized The more frames you record the more accurate the computed calibration will be However more frames also cause the computation of the calibration parameters to take longer The SP1 supports the recording of up to 40 cali bration frames The recording of calibration frames can be simplified by activating the auto capture mode In this mode a new calibration frame is recorded in fix capture intervals You can enter the desired interval in the auto capture section and then press the start auto capture button If desired an audible sound can be played to signal the countdown and the recording of a new frame Auto capture mode can be stopped by pressing the stop auto capture button A small preview of each captured calibration frame is added to the captured frames section The frames are overlaid with the detected positions of the calibration board circles You can click any of the preview images to see the calibration frame at its full resolution An example for a calibration frame with a correctly d
33. rames you can record additional frames and restart the calibration process 8 8 Reviewing Calibration Results Once calibration has been performed you can inspect the calibration results on the review calibration page which is shown in Figure On the top of this page you can see a live preview of both cameras which have been rectified with the current calibration parameters Please make sure that corresponding points in the images of both cameras have an identical vertical coordinate By activating the display epipolar lines option you can overlay a set of horizontal lines on both camera images This allows for an easy evaluation of whether the equal vertical coordinates criterion is met An example for a left and right input image with overlaid epipolar lines is shown in Figure In the quality information section you can find the average reprojection error This is a measure for the quality of your camera calibration with lower values indicating better calibration results Please make sure that the average reprojection error is well below 1 pixel All computed calibration parameters are displayed in the calibration data section These parameters are 24 8 8 Reviewing Calibration Results 8 CONFIGURATION S P T Review Calibration Calibrated Camera View Display epipokwy ines 216 ps Quality Information Average reprojection eror 0 069 Calibration Data amp 01881191450905536 02 0 312276591975551986 02 M 0 80350
34. ration data section This allows you to easily import the calibration data into your own applica tions Furthermore you can save the calibration data to your PC and reload it at a later point by using the upload calibration data section This allows you to switch between different cameras or optics without repeating the calibration process You can also perform a reset of the calibration data by pressing the reset calibration button In this case image rectification is disabled and the unmod ified image data is passed on to the stereo matching algorithm Use this option when selecting the already rectified virtual example camera as explained in Section 26 8 9 Network Settings 8 CONFIGURATION SP1 Chromium SP1 x u a lt 3 192 168 10 10 netw LO S P 1 Network Settings System Status IP Settings Notice Please make sure that you write down the new address when changing the IP address Otherwise you might not be able to communicate with the device Processing Settings IP Address 192 168 10 10 Subnet Mask 255 255 255 0 te MEEN Calibrate Cameras Change Review Calibration Network Settings Communication Settings Hel TCP UDP Recommended Target IP 224 0 0 1 Port 7681 Change Figure 21 Screenshot of configuration page for network settings 8 9 Network Settings The network settings page which is displayed in Figure is used for con figuring all network related parameters In the IP settings section y
35. rigger 1 can only be enabled if trigger 0 is also enabled The frequency is forced to the same value as trigger 0 However it is possible to specify a time offset which is the delay from a rising edge of trigger 0 to a rising edge of trigger 1 Furthermore trigger 1 can have a pulse width that differs from trigger 0 If PPS synchronization is enabled trigger 1 cannot be used 8 6 Time Synchronization The time synchronization page which is shown in Figure can be used to configure two possible methods for synchronizing the SP1 s internal clock As explained in Section the internal clock is used for timestamping captured camera frames The first possible option is to synchronize with a time server using the Network Time Protocol NTP up to version 4 In this case the SP1 synchro nizes its internal clock to the given time server using Coordinated Universal Time UTC The accuracy of the time synchronization depends on the la tency of your network and time server If NTP time synchronization is active synchronization statistics are displayed in a dedicated status area When activating Pulse Per Second PPS synchronization the internal clock is set to 0 whenever a synchronization signal is received as explained in Section 6 6 Please note that PPS synchronization and the trigger 1 output cannot be active at the same time as they both use the same physical pin 20 8 7 Camera Calibration 8 CONFIGURATION SP1 Chromium SP1 xo
36. s you to enter the calibration board details manually The first dimension of the pattern size is the number of circles in one grid column This number must be equal for all columns of the circles grid The number of circles per row is allowed to vary by 1 between odd and even rows The second dimension is thus the sum of circles in two consecutive rows All downloadable default calibration patterns have a size of 4 x 11 The final parameter that you have to enter when using a custom calibration board is the circle spacing This is the distance between the centers of two neighboring circles The distance must be equal in horizontal and vertical direction for all circles 21 8 7 Camera Calibration 8 CONFIGURATION SF za oc 52 168 10 10 S P 1 Calibrate Cameras ECEN camera preveu Calibration Board Caibroi Panem size Control Capao singe rana Call Fost Capured hames 0 Auto Capture Figure 16 Screenshot of configuration page for camera calibration Figure 17 Calibration board used by SP1 22 8 7 Camera Calibration 8 CONFIGURATION Figure 18 Example calibration frame with detected calibration board 8 7 2 Recording Calibration Frames Once the calibration board settings have been set you can begin recording calibration frames A live preview of both cameras is displayed in the camera preview area Make sure that the calibration board is fully visible in both camera images and then
37. signement for trigger output Please note that the SP1 has not been designed to handle high throughput bi directional network traffic Network image transfer is thus only provided for testing purposes The bi directional traffic flow can cause dropped pack ets which leads to dropped frames for UDP connections or low transfer speeds for TCP connections This problem can be mitigated when transferring image data at a lower frame rate It is recommended to always use TCP as underlying network protocol when performing network image transfer The SpCom sample application described in Section can be used for transferring a set of locally stored images to the SP1 An example command line for using SpCom for image transfer is as follows gt spcom t f30 h 192 168 10 10 s input dir In this case a TCP connection to the SP1 with IP address 192 168 10 10 is established and image data is streamed with a rate of 30 frames per second The image data is read from the directory input dir in alphabetical order Please make sure that images from the left camera always appear before the corresponding image from the right camera when sorted alphabetically 6 5 External Trigger For stereo matching it is important that both cameras are synchronized mean ing that both cameras record an image at exactly the same point of time Many industrial cameras already feature the ability to synchronize themselves by having one camera produce a trigger sign
38. spective software licenses can be obtained from the open source in formation website Some of these components may contain code from other open source projects which may not be listed here For a definitive list please consult the respective source packages The following individuals have requested to be named as author or co author of one of the included open source projects The PHP Group Google Inc The Android Open Source Project Red Hat Incorporated University of California Berkeley David M Gay Christopher G Demetriou Royal Institute of Technology Alexey Zelkin Andrey A Chernov FreeBSD S L Moshier Citrus Project Todd C Miller DJ Delorie Intel Corporation Henry Spencer Mike Barcroft Konstantin Chuguev Artem Bityuckiy IBM Sony Toshiba Alex Tatmanjants M Warner Losh An drey A Chernov Daniel Eischen Jon Beniston ARM Ltd CodeSourcery Inc MIPS Tech nologies Inc Addison Wesley Advanced Micro Devices Inc Alexander Chemeris Alexan der Neundorf Alexandre Benoit Alexandre Julliard Andreas Dilger Andrei Alexandrescu Andrey Kamaev Annecy le Vieux and GIPSA Lab Argus Corp Arren Glover AWare Sys tems Baptiste Coudurier Chih Chung Chang Chih Jen Lin Christopher Diggins Clement Boesch Danny Mayer David G Lowe Edward Rosten Enrico Scholz EPFL Signal pro cessing laboratory 2 Ethan Rublee ETH Zurich Autonomous Systems Lab ASL Fabrice Bellard Farhad Dadgostar Frank D Cringle Glenn Randers P
39. ub pixel resolution 4 bits 1 16 pixel Supported image size 640 x 480 pixels Post processing Consisteny check uniqueness check gap interpolation noise reduction speckle filtering 3 3 3 Image Rectification 4 MECHANICAL SPECIFICATIONS 75 e Power Busy Trigger 105 c a Figure 1 a Front b rear c side and d top view of SP1 with dimensions in millimeters 3 3 Image Rectification Horizontal displacement 31 to 31 pixels Vertical displacement 31 to 31 pixels Interpolation Bilinear 4 Mechanical Specifications 4 1 Dimensions Figures La to Id show the SP1 as seen from front rear side and top The provided dimensions are measured in millimeters 4 2 Mounting The SP1 is optionally available in a surface mountable version This version features a mounting plate that is attached to the bottom side of the housing The four through holes on the mounting plate can be used for mounting the SP1 onto a flat surface The dimensions of this mounting plate are shown in millimeters in Figure 3 The through holes are compatible to screws with an M4 ISO metric thread 4 2 Mounting 4 MECHANICAL SPECIFICATIONS Figure 2 SP1 with attached mounting plate 95 en o Figure 3 Mounting plate dimensions in millimeters 5 PHYSICAL INTERFACES Trigger port Busy LED Ethernet port um B Power Busy Trigger Cx M 0 2 amp j b
40. ust the right camera s settings accordingly 8 4 3 Recommended Settings for Point Grey Cameras When using Point Grey USB cameras we recommend that you use the follow ing settings e Enable external trigger and select the desired trigger source e If the cameras are connected to the SP1 s trigger port set the trigger 18 8 5 Trigger 8 CONFIGURATION polarity to active high Otherwise set the polarity according to your trigger source e Set the trigger mode to 1 recommended or 0 Always use mode 14 if the internal camera frame rate is not significantly higher than the trigger frequency e If an automatic adjustment of exposure sharpness and shutter is desired then please apply the following settings For the left camera set the mode of exposure sharpness shutter and gain to auto For the right camera set the mode of exposure sharpness shutter and gain to synchronize to left camera 8 4 4 Recommended Settings for IDS Imaging Cameras When using cameras from IDS Imaging we recommend that you use the fol lowing settings e If the cameras shall be triggered by the SP1 trigger output Due to voltage levels the SP1 trigger output needs to be connected to a camera GPIO pin rather than the trigger input The connected GPIO pin needs to be selected as trigger source The trigger mode shall be set to hardware trigger rising edge e Set the camera frame rate to at least twice the trigger fr
41. ving the processing results of the SP1 over a computer network Furthermore the API also allows for the transmission of image pairs It can thus be used for emulating the SP1 when performing systems development or for transmitting image data to the SP1 when using network image input The transmitted processing results always consist of two images Usually these are the rectified left camera image and the computed left camera disparity map If configured however the SP1 can also provide the raw camera images or the rectified images of both cameras see Section B 4 28 9 2 ImageTransfer Example 9 API USAGE INFORMATION Original and rectified camera images are always transmitted with a bit depth of 8 bits per pixel The disparity map is always transmitted with a bit depth of 12 bits Inside the library the disparity map is inflated to 16 bits to allow for more efficient processing The API provides three classes that can be used for receiving and trans mitting image pairs e ImageProtocol is the most low level interface This class allows for the encoding and decoding of image pairs to from network messages You will have to handle all network communication yourself e ImageTransfer opens up a network socket for sending and receiving im age pairs This class is single threaded and will thus block when receiving or transmitting data e AsyncTransfer allows for the asynchronous reception or transmission of image pairs This
42. y d can be reconstructed as follows j T u 1 x j y y gt y with Q w PA d PA zZ w 1 An efficient implementation of this transformation is provided with the available API see Section 9 4 The SP1 computes disparity maps with a resolution that is below one pixel Disparity maps have a bit depth of 12 bits with the lower 4 bits of each value representing the fractional disparity component It is thus necessary to divide each value in the disparity map by 16 in order to receive the correct disparity magnitude The SP1 applies several post processing techniques in order to improve the quality of the disparity maps Some of these methods detect erroneous disparities and mark them as invalid Invalid disparities are set to OxFFF which is the highest value that can be stored in a 12 bit disparity map In Figure 9b invalid disparities have been depicted as black 7 3 Timestamps and Sequence Numbers Each pair of rectified left camera image and disparity map which is transmitted by the SP1 also includes a timestamp and a sequence number The timestamp is measured with microsecond accuracy and it identifies the time at which a frame has been received by the SP1 It does not incorporate camera internal delays and delays caused by image transfer For images that are received over the network as noted in Section 6 4 the timestamp and the sequence number are both copied As explained in Sections 6 6 and 8 6 it is possi
Download Pdf Manuals
Related Search