Home

AM4001 IPMI: MMC User Manual

image

Contents

1. ipmitool 1 for the download of an image file e g ipmitool fru write 0 lt file name gt All IPMI interfaces which are offered by ipmitool are usable KCS interface IPMB bus LAN This allows local upgrade or remote upgrade Note that the KCS interface is only usable on a powered payload 8 5 E Keying E Keying has been defined in the AMC 0 R1 0 Specification to prevent module damage prevent misoperation and verify bay connection compatibility The FRU data contains the AMC point to point connectivity record as described in Section 3 9 2 of the AMC 0 R1 0 specification The Set Get AMC Port State IPMI commands defined by the AMC 0 specification are used by the carrier for either granting or rejecting the E keys Page 32 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide The PCI E reference clock may be generated locally by the module default configuration or externally common external reference clock connected via the AMC connector The standard FRU E Keying data for the AM4100 contains an AMC Link Descriptor for the PCI E clock 8 6 PCI Express Clock Source The common reference clock provided by the carrier may be slightly modulated Spread Spectrum Clock SSC Standard FRU E Keying data for AM4100 contain two AMC Link Descriptors for the PCI E channel first the one for SSC and second the one for none SSC If an external clock is available the carrier s IPMC assigns its presence
2. IPMI Firmware User Guide for the AM4100 Module Manual ID 36725 Rev Index 1 0 Firmware SK IPMI AM4100 EKS Idx 0100 October 15 2007 G kontron IN IPMI Firmware User Guide AM4100 r 1 Copyright Copyright 2007 Kontron AG Kontron Modular Computers makes no representations or warranties with respect to the contents or use of this manual and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose Kontron Modular Computers makes no representations or warranties with respect to this embedded Linux package and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose Permission is granted to make and distribute verbatim copies of this manual provided that the copyright notice and this permission notice are preserved on all copies Permission is granted to copy and distribute modified versions of this documentation under the conditions for verbatim copying provided also that the entire resulting derived work is distributed under the terms of a permission notice identical to this one Permission is granted to copy and distribute translations of this documentation into another language under the above conditions for modified versions The PICMG and CompactPCI names and the PICMG CompactPCI ATCA and AdvancedTCA logos are registered trademarks and AdvancedMC is a trademark of the PCI Industrial C
3. Extensions Page 16 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide 6 2 2 OEM Commands and Extensions 6 2 2 1 Get Device ID Command with OEM Extensions Get Device ID command with OEM extensions 00h App 06h 01h Byte Data Field Request Data Response Data 1 Completion Code 2 12 Regular Get Device ID Command response fields 13 Release index e g 101 denotes release index 1 01 Please note Some tools offer this as SDR version 14 Module Geographical Address site number 1 8 Module in AMC bay A1 A2 A3 A4 B1 B2 B3 B4 or in UTCA slot 1 8 with bus addresses 72h 74h 76h 78h 7ah 7ch 7eh 80h 9 12 Module in uTCA slot 9 12 with bus addresses 82h 84h 86h 88h 0 gt 12 Module position is not in range The IPMB L bus is switched off ref to 8 3 15 Reserved 16 Reserved ID 36725 Rev 1 0 2007 Kontron AG Page 17 IPMI Firmware User Guide AM4100 6 2 2 2 Set Firmware Parameters The command below permits the selection of interrupts to be used during KCS communication Set Firmware Parameters 03h OEM 3Eh 05h Request data 1 Reserved B4h 2 Reserved 90h 3 Reserved 91h 4 Reserved 8Bh 5 Cmd Flags 6 2 Reserved 1 Ob get only 0 Ob do not reset 6 Operating Modes 7 5 Reserved 4 Reserved 3 1 Reserved 0 Reserved 1b 7 IRQ number FFh do
4. Field Request data 1 Control ID 00h Firmware MMC Flash selection 9Dh Firmware Boot Order Configuration Response data 1 Completion code 2 Current Control State refer to 6 2 2 3 Set Control State Firmware Hub Boot Order Page 20 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide 6 2 2 5 Set or Get FRU Sub Device Number For the specific organization of the AM4100 s FRU data in sub devices refer to 8 4 2 Board Specific FRU Data Organization For e g FRU data updates it is possible to temporarily override the internal FRU sub device selection by the OEM command below Every time when officially FRU 0 data is accessed in reality the data of the selected sub device is accessed Set or Get FRU Sub Device Number 00h OEM 3Eh 22h Byte Data Field Request data 1 Sub device number 00h 03h Set these numbers to select the FRU 0 sub devices 0 3 OxFE Set this code to get the number of the currently selected FRU 0 sub device OxFF Set this code to reset the number of the currently selected FRU 0 sub device to the hardware settings Dip switch or solder jumper refer to the hardware manual Response data 1 Currently selected FRU 0 sub device number Please note This command selects only the storage of the currently used FRU data and NOT the board s operation mode Settings changed with this command are volatile and only valid until the n
5. Platform Alert 24h 0000h Aggregate states Y Digital discrete 03h 0000h power temperatures 0003h etc Visualization by the Health LED MMC Reboot Platform Alert 24h 0002h MMC reboot active N Digital discrete 03h 0000h state Is asserted 0003h during boot time Ver change Version change 2Bh 0002h Version change N Digital discrete 03h 0000h 0003h ID 36725 Rev 1 0 O 2007 Kontron AG Page 27 IPMI Firmware User Guide AM4100 r 7 1 2 OEM Event Reading Types OEM OEM SENSOR EVENT READING DESCRIPTION TYPE CODE TYPE CODE Firmware Info 1 COh 70h Internal Diagnostic Data Firmware Info 2 COh 71h Internal Diagnostic Data Board Reset C4h 6Fh Sensor Event sensor type specific specific Offset 00h Reserved 01h HwPowerReset 02h PCIReset 03h HwWatchDogReset 04h SoftReset 05h Reserved 06h ColdReset 07h IPMICommand 08h Reserved 09h Reserved OAh BMCWatchdog IPMBL State C3h 6Fh Sensor Meaning sensor type specific discrete State 08h IPMB L running others IPMB L not running Post Value C6h 6Fh Sensor Meaning sensor type specific discrete State Bits 7 0 Post Value read from host I O port 80h Bits 15 8 Reserved Page 28 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide OEM OEM SENSOR EVENT READING DESCRIPTI
6. T r 2 gt FN 3 De 0005 0 EEE EE 4 Ay Totroduetonss ussnadqneasniisnnnaneanansmannneiaanaddpageuniisvdadthannpsad 6 4 1 Acr nym DIS sdGS 6 4 2 Related PVCU ATOM Gere 7 5 WENO GUC TRO agi sasscni ch seevanseaceataisavewaehaqncindn dbeuiaaphaneuatabacacaaghadeitaahoheeiaphaterdaenancaaiabaiecernsiecioninnts 8 5 1 IPMI in AdvancedMC AdvancedTCA Environment esnrnrnrnrennvrrnrrrnnrernrrrrrrernseennne 8 5 2 Module Management Controller Hardware cccccccccsseceeeceeeeeeeeceeeceeeeeeeeeeseeesaeens 8 6 MT eee 9 6 1 Key 0 EEE KE 9 6 2 Supported IPMI Commands isinsin ant ud 10 6 2 1 Standatrd NG 0 1 RR NE EE 10 6 2 2 OEM Commands and Extensions ccc2sccasaccpersiecdestassterscensoiercancadeagseentexeantantavenact 17 6 2 2 1 Get Device ID Command with OEM Extensions reenvvrenvrrrnrerrrrrrsrenne 17 6 2 2 2 S t Firmware Parameters sinsvissessriassicaniesiai eassa aa Ei aas 18 6 2 2 3 Set Control State Firmware Hub Boot Order rrrrrnnvvrrnnvvrrnnvrrnnnvnrnnn 19 6 2 2 4 Get Control State Firmware Hub Boot Order srrronvrronrrrnnrerrrnrrnnrrnnne 20 6 2 2 5 Set or Get FRU Sub Device Number ennonrennvenvvsserevrsersenrersvensesnesenseeree 21 6 2 2 6 OEM Module Quiescence Feedback uunsannssanmmsnmnessismt imnnid 22 7 Sensors Implemented on the AIA 00 aaasesaassaaed annet 24 7 1 1 Sensor EEE EEE EN E EEEE 25 7 1 2 OEM Event Reading Types eeessessesseessessesseessessesseessessosseesseesosseesseesseseess
7. actions For this the IPMI command ChassisControl may be used to cause a power cycle or better the first power up To get a perfect synchronization of MMC and payload it is needed that a payload power on event is performed after management power on Otherwise no automatic reboot of the payload after a boot error is performed Page 36 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide 12 OS Support Tools 12 1 1 Linux Normally all drivers and kernel modules needed for communication between the payload sided software and the MMC firmware via the KCS interface come with the distribution Newest sources can be downloaded from http openipmi sourceforge net There may be downloaded the OpenIPMI project as well The OpenIPMI library package includes some applications and the needed libraries One of the applications is ipmicmd which makes it possible to send and receive raw IPMI sequences another the ipmi_ui provides a higher level interface and thus it does not require deep IPMI knowledge from the user side Another very useful all in one tool is the ipmitool http ipmitool sourceforge net It provides a user friendly interface to many IPMI features and extensions for example to PICMG LED control and for the upgrade of the MMC s firmware ipmitool fwum There is a Kontron Modular Computers IPMI ToolKit which contains some tools for the customer for the monitoring an
8. during E keying by Set AMC Port State commands As a result of any SSC or non SSC selection the module will connect to the external clock by setting a signal This connection will be activated by the next payload power on and can t be switched off until the next payload power off ID 36725 Rev 1 0 2007 Kontron AG Page 33 IPMI Firmware User Guide AM4100 r 9 NetBootLoader Flash Selection When the AM4100 s payload starts the first code to be executed is the NetBootLoader There are two Flash devices number 0 and 1 which may contain different NetBootLoader code Which one of them will be selected for the next boot process on depends on The position of an on board DIP switch refer to the AM4100 User Guide b The decision of the MMC to logically modify toggle or not to modify this preselection The MMC s current decision is stored in its EEPROM and is valid from the next boot process on c The contents of a user writeable register refer to the AM4100 User Guide There are two reasons for the MMC to adjust its decision or not The first cause is an OEM IPMI command Set Control State The second cause is a failing boot process Please refer to the chapters below 9 1 1 Boot Flash Selection by OEM IPMI Command The OEM IPMI command Set Control State refer to 6 2 2 3 adjusts whether the MMC modifies the switch based Flash selection or does this not from the next boot process on The MMC stores th
9. state is entered and exited Please note that some carriers reset the MMC during hot swap That means that all settings have to be renewed after that _ ID 36725 Rev 1 0 2007 Kontron AG Page 23 IPMI Firmware User Guide AM4100 r 7 Sensors Implemented on the AM4100 The MMC includes many sensors for voltage or temperature monitoring and various others for pass fail type signal monitoring Every sensor is associated with a Sensor Data Record SDR Sensor Data Records contain information about the sensors identification such as sensor type sensor name sensor unit SDRs also contain the configuration of a specific sensor such as threshold hystheresis event generation capabilities that specifies sensor behavior Some field of the sensor SDR are configurable through IPMI v1 5 command and are set to built in initial value Finally one field which is the sensor owner must reflect the module addresses that allow the AMC Carrier to identify the owner of the SDR when it is scanned from the module management controller and merged within the AMC Carrier Device SDR repository From IPMI perspective the MMC is set up as a satellite management controller SMC It does support sensor devices and uses the IPMI static sensor population feature of IPMI v1 5 All SDRs can be queried using Device SDR commands to the MMC The sensor name in its SDR has a name prefix which after module insertion is automatically adapted to the physical position of t
10. versions concerning its GbE interfaces e 2 GbE on the front panel 2 GbE on the AMC connector e 4 GbE on the AMC connector The version is fixed during the manufacturing process Each of these two hardware versions can operate in four different ways e PCIe Root Complex sub device 0 e PCIe End Point sub device 1 e sRIO Host sub device 2 e sRIO Agent sub device 3 The code for the selection of one of these four modes of operation is adjustable by DIP switches or solder jumpers please refer to the hardware manual The 4 kB wide non volatile FRU 0 data storage of the board is internally divided into four 1 kB wide sub devices 0 3 For every mode of operation the appropriate sub device is selected automatically but is offered to the outside always as FRU 0 For FRU data updates it is possible to temporarily override the internal FRU sub device selection by the OEM command Set or Get FRU Sub Device Number in chapter 6 2 2 5 Every time when officially FRU 0 data is accessed in reality the data of the selected sub device is accessed 8 4 3 Download Normally the user doesn t need to download the FRU information because the module is shipped with the data which describes best the module s capabilities and requirements Remember that FRU information might be damaged by a write access to the IPMI FRU Device using standard IPMI commands If needed the standard way to download FRU information to the module is to use the open tool
11. 280h near AMC connector 1818h Temp CPU Out Temperature 01h 0280h Outlet temperature Y Threshold 01h 3280h near CPU 1818h CPU status Processor 07h Sensor 0002h CPU thermal alarm N specific 6Fh 0002h sensor 0002h POST Value OEM Post Value C6h 0000h POST Value from N OEM 78h 0000h host I O port 80h O0FFh FWHO Boot Err Boot Error 1Eh Sensor 0008h Firmware Hub 0 Y specific 0008h Boot Flash 0 boot 0008h error FWH1 Boot Err Boot Error 1Eh Sensor 0008h Firmware Hub 1 Y specific 0008h Boot Flash 1 boot 0008h error Pwr Good Power supply 08h OEM 0000h Status of all power N 77h 0000h lines 18B7h Pwr Good Evt Power supply 08h OEM 0000h Power fail events for Y 77h 18B7h all power lines 18B7h Page 26 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide SENSOR TYPE ASS Mask CODE I Deass Causes red EVENT READING Hebe DESCRIPTION Helge TYPE CODE Sarding EELS Lan Front Lk LAN 27h Sensor 0000h LAN Front link status N specific 0000h or LAN port 8 status 0003h if supported by the module Lan Front1 Lk LAN 27h Sensor 0000h LAN port 9 status if N specific 0000h supported by the 0003h module Lan AMCO Link LAN 27h Sensor 0000h LAN port 0 status N specific 0000h 0003h Lan AMC1 Link LAN 27h Sensor 0000h LAN port 1 status N specific 0000h 0003h Health Error
12. Bridge O No Other Bridge Commands o Error Report ICMB Bridge FFh O No ID 36725 Rev 1 0 2007 Kontron AG Page 15 IPMI Firmware User Guide AM4100 AdvancedTCA 10 PICMG M 3 0 Table Get PICMG Properties 3 9 PICMG 00h M Yes Get Address Info 3 8 PICMG Olh N A Get Shelf Address Info 3 13 PICMG 02h N A aS FRU Control 3 22 PICMG 04h M Yes 2 Get FRU LED Properties 3 G M Yes Get LED Color Capabilities M Yes Set FRU LED State M Yes Get FRU LED State 3 27 PICMG 08h M Yes Set IPMB State 3 51 PICMG 09h N A PICMG Set FRU Activation Policy OAh N A Get FRU Activation Policy PICMG OBh N A Set FRU Activation PICMG OCh N A Get Device Locator Record ID PICMG ODh M Yes Set Port State PICMG OEh N A Get Port State PICMG OFh N A Compute Power Properties PICMG 10h N A Set Power Level PICMG 11h N A Get Power Level PICMG 12h N A Renegotiate Power PICMG 13h N A Get Fan Speed Properties PICMG 14h N A Set Fan Level PICMG 15h N A Get Fan Level PICMG 16h N A Bused Resource PICMG 17h N A Get IPMB Link Info 18h N A mw Be 1 1 Table Set AMC Port State Table 3 27 PICMG 19h O Yes Get AMC Port State Table 3 28 PICMG 20h O Yes 1 Response byte 2 hours byte 3 minutes after module start Bytes 4 6 void 2 Request byte 3 only 04h quiesced implemented 3 Has oem extensions Please refer to 6 2 2 1 Get Device ID Command with OEM
13. No SOL Activating 26 1 Transport 20h O Yes Get SOL Configuration Parameters 26 2 Transport 21h O Yes Set SOL Configuration Parameters 26 3 Transport 22h O Yes Page 14 O 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide Bridge Management Commands ICMB o Get Bridge State ICMB Bridge 00h O No Set Bridge State ICMB Bridge Olh O No Get ICMB Address ICMB Bridge 02h O No Set ICMB Address ICMB Bridge 03h O No Set Bridge Proxy Address ICMB Bridge O No Get Bridge Statistics ICMB Bridge 05h O No Get ICMB Capabilities ICMB Bridge 06h O No Clear Bridge Statistics ICMB Bridge 08h O No Get Bridge Proxy Address ICMB Bridge O No Get ICMB Connector Info ICMB Bridge OAh O No Get ICMB Connection ID ICMB Bridge OBh O No Send ICMB Connection ID ICMB Bridge OCh O No Discovery Commands ICMB Oo Prepare For Discovery 10h O No Get Addresses 11h O No Set Discovered 12h O No Get Chassis Device ID 13h O No Set Chassis Device ID 14h O No Bridging Commands ICMB o Bridge Request ICMB Bridge 20h O No Bridge Message ICMB Bridge 21h O No Event Commands ICMB o Get Event Count ICMB Bridge 30h O No Set Event Destination ICMB Bridge O No Set Event Reception State ICMB Bridge 32h O No Send ICMB Event Message ICMB Bridge O No Get Event Destination ICMB Bridge 34h O No Get Event Reception State ICMB Bridge 35h O No OEM Commands for Bridge NetFn o OEM Commands ICMB
14. ON In dd 070 2 9 TYPE CODE i e for 77h Sensor Event Power Good specific Offset Power Good Event Oh 12V good th 5V good 2h 3V3 good 3h Reserved 4h 1V8 good 5h 1V5 good 6h Reserved 7h vecCore good 8h Reserved 9h 1V1 good Ah Reserved Bh 3V3IPMI good Ch OV9 good Hot swap sensor F2h 6Fh Sensor Event sensor type specific specific Offset 00h Handle close 01h Handle open 02h Quiesced ID 36725 Rev 1 0 2007 Kontron AG Page 29 IPMI Firmware User Guide AM4100 r 8 Firmware Code 8 1 Structure and Functionality MMC firmware code is organized into boot code and operational code both of which are stored in a flash module Upon an MMC reset the MMC executes the boot code and performs the following e Self test to verify the status of its hardware and memory e Performs a checksum of the operational code e Communicates with the Firmware Upgrade Manager FWUM in order to inform the MCC watchdog that the actual MMC firmware is suitable for execution Upon successful verification of the operational code checksum the firmware will jump to the operational code Operational code is upgradeable in the field 8 2 Firmware Upgrade The standard way to upgrade the MMC s operational code is to use the open tool ipmitool 1 together with an image file Ipmitool allows the downloading ipmitool download and activation ipmitool upgrade of the new operational code and sav
15. alth state of a running powered module Behavior Blinking Module MMC is running showing its heart beat Pulsing KCS interface active Off Module MMC not running Any action green No health error detected refer to sensor Health Error 7 1 1 Any action red Health error detected Index in Get Set LED State commands 2 Page 38 2007 Kontron AG ID 36725 Rev 1 0
16. and e When the MMC reads this flag it reports to the carrier controller that the payload shut down has finished so that the Hot Swap processing can be finished e By default the MMC waits endlessly for this flag If an endless wait is to be avoided it is necessary to set a timeout time For the setting of the timeout refer to 6 2 2 6 OEM Module Quiescence Feedback After this timeout time the MMC reports to the carrier controller that the payload shut down has finished so that the Hot Swap processing can be finished ID 36725 Rev 1 0 2007 Kontron AG Page 35 IPMI Firmware User Guide AM4100 r 11 Module Not Running on an ATCA Carrier If you run the AM4100 in an environment which is not compatible to an ATCA carrier you will have to consider the following terms and conditions The MMC s management power must permanently be available after system start An ATCA compatible E Keying process must be performed to cause the module to select the external PCIe clock signal After the E Keying a payload power on 12V has to be performed to activate this selection For this the IPMI command ChassisControl may be used to cause a power cycle or better the first power up There are other signals which must be set by the MMC according to the selection of run modes mode switches These signals only become valid after a payload power on event So at least perform a payload power on reset after the MMC has done the start up
17. d Reset OEM C4h Sensor 04DEh Board reset event Y specific 0000h Board 3 3V Voltage 02h Threshold 2204h Board 3 3V supply Y 01h 2204h 1212h Board 3 3vIPM Voltage 02h Threshold 2204h AMC Management Y 01h 2204h I Power MP 3 3V 1212h Board 12 0v Voltage 02h Threshold 2204h AMC Payload Power Y 01h 2204h PWR 12V 1212h Board 1 8V Voltage 02h Threshold 2204h Board 1 8V supply Y 01h 2204h 1212h Board Vtt0 9V Voltage 02h Threshold 2204h DDR termination Y 01h 2204h supply 1212h ID 36725 Rev 1 0 O 2007 Kontron AG Page 25 IPMI Firmware User Guide r AM4100 SENSOR TYPE fee CODE I Deass Causes red EVENT READING Bed DESCRIPTION Her TYPE CODE ang MER Board 5 0V Voltage 02h Threshold 2204h Board 5V supply Y Oth 2204h 1212h Board 1 5V Voltage 02h Threshold 2204h Board 1 5V supply Y Oth 2204h 1212h Board 1 1V Voltage 02h Threshold 2204h Board 1 1V supply Y Oth 2204h 1212h Temp CPU Temperature 01h 0280h CPU die temperature Y Threshold 01h 3280h 1818h Temp CPU In Temperature 01h 0280h Inlet temperature Y Threshold 01h 3280h near CPU 1818h Temp AMC Out Temperature 01h 0280h Outlet temperature Y Threshold 01h 3280h near AMC connector 1818h Temp AMC In Temperature 01h 0280h Inlet temperature Y Threshold 01h 3
18. d the maintenance of some IPMI functionalities The following command line tools are included in the IPMI Tool Kit e frum display and modification of FRU data e temptool selection display and storage of temperature and voltage sensor values All these tools are OpenIPMI based This toolkit is available on the AMC Kit CD If this CD was not shipped with your module then please contact our support support kontron modular com Please refer to the manual MAN LIN IPMI TOOLKIT 0103 pdf being included in the package ID 36725 Rev 1 0 2007 Kontron AG Page 37 IPMI Firmware User Guide AM4100 m 13 IPMI Communication LEDs There are three IPMI communication LEDs on the face plate LED 0 Hot Swap Color Blue Position Bottom right Labeled Meaning Indicates the Hot Swap state of the inserted and powered module Meaning On Module may be extracted Blinking Hot Swap active don t extract Off Module in normal operation Don t extract Index in Get Set LED State commands 0 LED 1 Out Of Service Color Red Position Top left Labeled by a crossed out plus sign Meaning Indicates the Out Of Service state of the powered module Behavior Solid red module out of service Off module powered running Pulsing Traffic on the IPMB L bus Index in Get Set LED State commands 1 LED 2 Health Color Green red Position Top right Labeled by a plus sign Meaning Indicates the He
19. ee 28 5 Fre rss 30 8 1 Struct re and Functondiiyusaassjasmaemauatbaiais minsnnnt sjaa 30 AE vn OG RE NS 30 8 3 Firmware Module Identification ss casiessaccsisteceanczerdensentwandieeacene seheancatsansentexesnacioatenes 31 84 FRU Te Fora sssrinin tase ean i eaan r EE E EAEE EA iaae EEA 31 8 4 1 Suu and Firer essorer ieissar ossei inia 31 8 4 2 Board Specific FRU Data Organization seesnoronvronvrrnnrrrrrrerrrrernnernvresrsrsrnennn 32 8 4 3 Downloadi ertse iernat nE eee 32 Page 4 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide MEH 0 EE eanensanasualdailesn est niwussensateniasesun hess dvecsisies 32 8 6 PCI Express Clock SoUebuanusasvasem nedsett E ER EEEE SERA EEs 33 9 NetBootLoader Flash See 34 9 1 1 Boot Flash Selection by OEM IPMI Command rasrrrnnvernnnrrnrrenvrrsnrnerrnrenseennee 34 9 1 2 Automatic Boot Flash Selection during the Boot Process rrasrrarnrrrrnnvrnvernnne 34 10 Hot SW AD erucdsewoenusearostenasasiesuadumnesueutiinacaeriaunesesuidesatsimesdqentianspasinstenunsidentinwesdsadivessoucenses 35 11 Module Not Running on an ATCA Carrier ssoronnrsnnvesnrnrersrsnsrsnnrrsnrersrrsensessnrssnrsssrneenn 36 12 OS Support EE EE EE EE 37 SEE das EAr E Ri 37 13 IPMI Communication LEDS siiscciass sascaasssienaadpacccavinsacseass aber tarnsacsaa tnaeexrnistaanassaberanseaeis 38 ID 36725 Rev 1 0 2007 Kontron AG Page 5 IPMI Firmware User Guide AM4100 r 4 introduction 4 1 Acr
20. epository Update Mode Run Initialization Agent ID 36725 Rev 1 0 Storage Storage Storage 2007 Kontron AG Page 13 IPMI Firmware User Guide AM4100 SEL Device Commands o Get SEL Info 40 2 Storage 40h O No Get SEL Allocation Info Storage 41h O No Reserve SEL Storage 42h O No Get SEL Entry Storage 43h O No Add SEL Entry Storage 44h O No Partial Add SEL Entry Storage 45h O No Delete SEL Entry Storage 46h O No Clear SEL Storage 47h O No Get SEL Time Storage 48h O No Set SEL Time Storage 49h O No Get Auxiliary Log Status Storage 5Ah O No Set Auxiliary Log Status Storage 5Bh O No LAN Device Commands o Set LAN Configuration Parameters Transport Olh O Yes Get LAN Configuration Parameters Transport 02h O No Suspend BMC ARPs Transport 03h O Yes Get IP UDP RMCP Statistics Transport 04h O Yes Serial Modem Device Commands o Set Serial Modem Configuration 25 1 Transport 10h O No Get Serial Modem Configuration 25 2 Transport 11h O No Set Serial Modem Mux 25 3 Transport 12h O No Set PPP UDP Proxy Transmit Data 25 5 Transport 14h O No Get PPP UDP Proxy Transmit Data 25 6 Transport 15h O No Send PPP UDP Proxy Packet 25 7 Transport 16h O No Get PPP UDP Proxy Receive Data 25 8 Transport 17h O No Serial Modem Connection Active 25 9 Transport 18h O No Callback 25 10 Transport 19h O No Set User Callback Options 25 11 Transport 1Ah O No Get User Callback Options 25 12 Transport 1Bh O
21. es an existing one The rollback to the formerly running operational code is possible as well ipmitool rollback All IPMI interfaces which are offered by ipmitool are usable KCS interface IPMB bus LAN This allows local upgrade or remote upgrade Files which contain an image of operational code have the module type AM4100 and the string FWUM in its name During the download process the currently running operational code operates as usual until the upgrade command is issued During the upgrade process the MMC is off line for about 45 seconds when the boot code is re organizing the firmware storage and afterwards starts the new operational code If this doesn t succeed after a time out the boot code performs an automatic rollback to the last working operational code Page 30 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide 8 3 Firmware Module Identification There are two ways to verify by means of IPMI that the MMC resides on an AM4100 1 The response on the IPMI command Get Device ID offers among others the following response data e Manufacturer ID 3A98h Kontron IANA ID e Device ID 04h H8S2166 e Product ID 4100 e Firmware Revision in bytes 4 5 depends on running firmware e The SDR revision in byte 13 OEM part of the response is a sub revision of the firmware revision 2 The Device ID String which can be found by reading the Management Controller Device Locato
22. ext reset of the MMC Example If you want to update the FRU data memory for mode 1 PCIe End Point you should proceed like this e Use the OEM command Set or Get FRU Sub Device Num and select sub device 1 e Use ipmitool 1 to write the FRU data via FRU 0 ipmitool fru write 0 lt filename gt e Use the OEM command Set or Get FRU Sub Device Num and select sub device Oxff to re activate the default sub device again ID 36725 Rev 1 0 2007 Kontron AG Page 21 IPMI Firmware User Guide AM4100 r 6 2 2 6 OEM Module Quiescence Feedback Overview Please refer to 10 Hot Swap This command provides support to control a graceful shut down of the AM4100 There is a Graceful Reboot and Shutdown Daemon grnsd for Linux included in the Linux BSPs being offered by Kontron It works as described in the following refer to Usage for a self written shut down daemon below Usage for a self written shut down daemon This command normally is used during a shut down to inform the MMC about the end of shutdown If a timeout time must be set to avoid endless waiting for the end of payload shutdown then the daemon sets this timeout in the command after system start with the set quiesce wait timeout bit set and the Quiesce wait timeout time lt gt 0 Afterwards the daemon has to call this command cyclically with the OS daemon present bit set When the MMC gets a FRU Control Quiesce request from the car
23. gement The carrier IPMI Controller PMC communicates with the MMC using the local IPMB IPMB L bus In an ATCA AMC environment it is the IPMC that actually turns on off module payload power However before the IPMC enables the module payload power various criteria must be satisfied by both the carrier and the module including power requirements and capabilities matching interfaces current module hot swap state and any other special conditions as specified by the Shelf Manager policy 5 2 Module Management Controller Hardware On the AM4100 module the MMC is implemented using the Renesas H8S 2166 controller with 512 kB of internal flash and 40 kB of RAM An additional 1 MB serial EEPROM chip provides redundant firmware image storage A separate 32 kB serial EEPROM chip is used for firmware private data and 4 kB FRU Inventory storage The MMC circuit implements two local Keyboard Style Interfaces KCS with interrupt support for communication with system side management software and the IPMB L bus for interconnection with the IPMC The MMC implements a wide range of sensors that permit the monitoring of e main power voltages 12V PWR 5V 3 3V 3 3V MP 1 5V 1 8V 0 9V DDR 1 1V e temperatures CPU die board inlet and outlet near CPU board inlet and outlet near AMC connector e Power Good IPMB link board reset post code boot error processor hot detection gt 105 C IPMB L state Health error IPMI watchdog e
24. he module in a carrier or in a UTCA chassis The format of this prefix is e in AMC bay 1 8 or pTCA slot 1 8 A1 A2 A3 A47 BL B2 B3 B4 e in uTCA slot 9 12 R9 Rar Rb Re Please note that in the case that the module is installed elsewhere then the IPMB L address of the module is unknown and the interface is off Module sensors that have been implemented are listed in the sensor list below Page 24 O 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide 7 1 1 Sensor List For OEM Kontron specific sensor types and codes in the following table please refer to chapter 7 1 2 SENSOR TYPE _ Ass Mask CODE I Deass Causes red EVENT READING Ce DESCRIPTION Henne ver TYPE CODE g POTOA EELS IPMI Info 1 OEM Firmware Info 1 0003h For internal use only N COh OEM 70h 0000h 7FFFFh IPMI Info 2 OEM Firmware Info 2 0003h For internal use only N COh OEM 71h 0000h 7FFFFh IPMI Watchdog Watch dog 23h 010Fh Watchdog 2 Y Sensor specific 0000h 010Fh FRU Agent OEM C5h Discrete 0140h FRU agent N OAh 0000h 0147h ModuleHotSwap OEM F2h Sensor 0007h Hot swap sensor N specific 0000h 0007h IPMBL State OEM C3h Sensor 000Fh State of IPMB L bus N specific 0000h 000Fh Storage Err Management subsystem 0002h Storage error N health 28h Sensor 0000h specific Boar
25. ion Capabilities 22 13 App 38h O Yes Get Session Challenge 22 15 App 39h O Yes Activate Session 22 17 App 3Ah O Yes Set Session Privilege Level 22 18 App 3Bh O Yes Close Session 22 19 App 3Ch O Yes Get Session Info 22 20 App 3Dh O Yes Get AuthCode 22 21 App 3Fh O No Set Channel Access 22 22 App 40h O Yes Get Channel Access 22 23 App 41h O Yes Get Channel Info 22 24 App 42h O Yes Set User Access 22 26 App 43h O Yes Get User Access 22 27 App 44h O Yes Set User Name 22 28 App 45h O Yes Get User Name 22 29 App 46h O Yes Set User Password 22 30 App 47h O Yes Activate Payload 24 1 App 48h O Yes Deactivate Payload 24 2 App 49h O Yes Get Payload Activation Status 24 4 App 4Ah O Yes Get Payload Instance Info 24 5 App 4Bh O Yes Set User Payload Access 24 6 App 4Ch O Yes Get User Payload Access 24 7 App 4Dh O Yes Get Channel Payload Support 24 8 App 4Eh O Yes Get Channel Payload Version 24 9 App 4Fh O Yes Get Channel OEM Payload Info 24 10 App 50h O No Master Write Read 22 11 App 52h O Yes Get Channel Cipher Suits 22 15 App 54h O No Suspend Resume Payload Encryption 24 3 App 55h O Yes Set Channel Security Keys 22 25 App 56h O No Get System Interface Capabilities 22 9 App 57h O No ID 36725 Rev 1 0 2007 Kontron AG Page 11 IPMI Firmware User Guide Chassis Device Commands Get Chassis Capabilities Chassis Get Chassis Status Chassis Chassis Control Chassis Chassis Reset Chassis Chas
26. is decision in the EEPROM 9 1 2 Automatic Boot Flash Selection during the Boot Process After each payload reset the MMC waits for a special message from the Boot Loader This message contains the checksum report i e it reports whether the boot Flash s checksum is right or wrong If the checksum is wrong or the message is not received within 20 seconds then the currently used NetBootLoader Flash is assumed to contain an invalid or a corrupted image In this case the MMC toggles again its decision to modify the preselection of the boot Flash from the next boot process on and stores this decision in its EEPROM Then it issues a Boot Error Invalid boot sector event for the failed NetBootLoader Flash and sets the appropriate sensor value sensor FWHx Boot Err x 0 1 Afterwards it causes a new payload reset When a timeout error is recognized and the count of boot errors exceeds 2 or when a checksum error is recognized and the count of boot errors exceeds 4 the MMC gives up i e causes no more payload resets after a boot error until the next payload power on event Page 34 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide 10 Hot Swap As a hot swappable field replaceable unit FRU the AM4100 also follows the same stringent carrier grade RASM feature set namely Reliability Availability Serviceability Maintainability When offered in combination with AdvancedTCA platforms TEM Telecom Equipment Man
27. not use interrupts OAh use IRQ10 OBh use IRQ11 Any other values Reserved Response data Completion code Cmd Flags Operating Modes IRQ number Bi w n gt Page 18 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide 6 2 2 3 Set Control State Firmware Hub Boot Order Set Control State Firmware Hub Boot Order 00h OEM 3Eh 20h Byte Data Field Request data 1 Control ID 00h MMC Flash selection 9Dh Firmware Boot Order Configuration Control State for MMC Flash selection These settings are stored in EEPROM and applied to logic each time the MMC detects power on 00h Flash selection by switch is not inverted 01h Flash selection by switch is logically inverted Please note that this selection will be automatically toggled during a failing boot process Other settings may additionally modify Control State for Boot Order Configuration 00h FFh Selected Boot Order Configuration being defined for the Net Boot Loader refer to the Net Boot Loader documentation These settings are stored in EEPROM and applied to logic each time the MMC detects power on Response data Completion code ID 36725 Rev 1 0 2007 Kontron AG Page 19 IPMI Firmware User Guide AM4100 6 2 2 4 Get Control State Firmware Hub Boot Order Get Control State Firmware Hub Boot Order 00h OEM 3Eh 2th Byte Data
28. omputer Manufacturers Group Intel is a registered trademark of Intel Corporation TC is a trademark of Phillips Semiconductors Linux is a registered trademark of Linus Torvalds All other trademarks registered trademarks and trade names are the property of their respective owners Page 2 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide 2 Revision History Manual Product Title IPMI Firmware User Guide for the AM4100 Module Manual ID Number Revision Index Brief Description of Changes Date of Issue Imprint Kontron Modular Computers GmbH may be contacted via the following MAILING ADDRESS TELEPHONE AND E MAIL Kontron Modular Computers GmbH 49 0 800 SALESKONTRON SudetenstraBe 7 sales kontron com D 87600 Kaufbeuren Germany For further information about other Kontron products please visit our Internet web site www kontron com Disclaimer Copyright 2007 Kontron AG All rights reserved All data is for information purposes only and not guaranteed for legal purposes Information has been carefully checked and is believed to be accurate however no responsibility is assumed for inaccuracies Kontron and the Kontron logo and all other trademarks or registered trademarks are the property of their respective owners and are recognized Specifications are subject to change without notice ID 36725 Rev 1 0 O 2007 Kontron AG Page 3 IPMI Firmware User Guide AM4100 r 3 Contents le
29. onym Definitions AMC BSP DMI FRU FWH 12C IPMB IPMB 0 IPMB L IPMC IPMI IOL KCS MMC MP PICMG PWR SDR SDRR SEL SMBIOS SMS SOL Page 6 Advanced Mezzanine Card Board Support Package Desktop Management Interface Field Replaceable Unit Firmware Hub Inter Integrated Circuit Intelligent Platform Management Bus AdvancedTCA shelf level IPMB Local on carrier IPMB that links the carrier IPMC with the MMCs of installed modules Intelligent Platform Management Controller located on AMC carrier Intelligent Platform Management Interface IPMI over LAN An MMC is accessed via LAN not IPMB Keyboard Controller Style Module Management Controller an IPMI controller located on the AMC module Management Power PCI Industrial Computer Manufacturer Group Payload Power Sensor Data Record Sensor Data Record Repository System Event Log System Management BIOS System Management Software designed to run under the OS Serial over LAN A serial interface is redirected by LAN using the RMCP protocol 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide 4 2 Related Documentation IPMI specifications http www intel com design servers ipmi spec htm e IPMI Intelligent Platform Management Interface Specification Second Generation v2 0 February 12 2004 part e 2 IPMI Platform Management FRU Information Storage Definition v1 0 Document Revision 1 1 September 1999 PICMG specifications h
30. ot image from a second boot device flash refer to 9 Graceful shutdown support refer to 10 Health LED shows heartbeat and pulses if a KCS interface is active The out of service OOS LED pulses when IPMB L is active refer to 13 If the carrier supplies an external clock 3 the module will use it Otherwise the module will use the internal PCIe clock refer to 8 6 ID 36725 Rev 1 0 O 2007 Kontron AG Page 9 IPMI Firmware User Guide AM4100 6 2 Supported IPMI Commands 6 2 1 Standard Commands Part of the command list in IPMI specification 2 0 M mandatory O optional Kontron IPMI 2 0 support Spec section EN END On MMC IPM Device Global M Commands M Yes 3 Get Self Test Results 20 4 App 04h Manufacturing Test On 20 5 App 05h BMC Watchdog Timer Commands o Page 10 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide BMC Device and Messaging Commands o Set BMC Global Enables 22 1 App 2Eh O Yes Get BMC Global Enables 22 2 App 2Fh O Yes Clear Message Flags 223 App 30h O Yes Get Message Flags 22 4 App 31h O Yes Enable Message Channel Receive 22 5 App 32h O Yes Get Message 22 6 App 33h O Yes Send Message 22 7 App 34h O Yes Read Event Message Buffer 22 8 App 35h O Yes Get BT Interface Capabilities 22 9 App 36h O Yes Get System GUID 22 14 App 37h O No Get Channel Authenticat
31. r Record SDR Type 12h contains the string AM4100 For a module being placed in bay B3 the Device ID String will be B3 AM4100 8 4 FRU Information 8 4 1 Structure and Functionality The MMC provides 4 kB non volatile storage space for FRU information Some of the data stored there like the Module Current Requirements record or E Keying information refer to AMC 0 specification for details are mandatory for module functionality in the ATCA AMC environment Refer to 2 Platform Management FRU Information Storage Definition document for details Please note that missing FRU information will prevent the AMC module from being accepted by the carrier controller during the Hot Swap process and the module will get no payload power Full low level access to read or write a module s FRU Information is provided by regular IPMI FRU Device commands Please be careful when writing FRU information directly using standard IPMI commands Damaging the FRU Information may lead to a non working payload To avoid this danger there exists a Kontron Linux tool frum refer to section 12 2 which allows to display and partially modify FRU data E g the frum tool makes it easy to modify Product Info Area fields like Product Version or Product Serial Number ID 36725 Rev 1 0 2007 Kontron AG Page 31 IPMI Firmware User Guide AM4100 r 8 4 2 Board Specific FRU Data Organization The AM4100 board comes in two different hardware
32. rier e g during a Hot Swap sequence it sets the quiesce request FRU Control bit in its command response After the daemon sees this bit set in the response it should shut down the system After having set the quiesce request FRU Control bit the MMC starts the timeout timer if a timeout time was defined and monitors the sleep signal line to recognize the sleep state which should be caused by the shut down When the MMC receives a command with the quiescence acknowledge bit set or the timeout timer has expired the MMC sends a Module Hot Swap event message to the carrier and in the following the payload power will be switched off If no daemon is announced as present If no command call announces that a daemon is present the MMC automatically uses the default wait timeout time 0 endless wait during the Hot Swap process until the daemon sends the Os ready using control bits 40h see below But if the wait timeout time was set to a value 1 255 by control bits 80h see below this wait timeout time will be used in any case while waiting for end of shutdown The waiting for the sleep process signal can immediately be ended by setting Os ready using control bits 40h see below Page 22 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide OEM Module Quiescence Feedback 00h OEM 3Eh 40h Byte Data Field Request data 1 Control bits 7 1b set quiesce wait timeou
33. sis Identify Chassis Set Chassis Capabilities Chassis Set Power Restore Policy Chassis Get System Restart Cause Chassis Set System Boot Options Get System Boot Options Chassis Chassis Get POH Counter Set Event Receiver Chassis Get Event Receiver Platform Event a k a Event Message PEF and Alerting Commands Get PEF Capabilities Arm PEF Postpone Timer Set PEF Configuration Parameters Page 12 2007 Kontron AG AM4100 ID 36725 Rev 1 0 AM4100 Sensor Device Commands Get Device SDR Info Reserve Device SDR Repository Get Sensor Reading Factors IPMI Firmware User Guide Set Sensor Hysteresis Get Sensor Hysteresis Set Sensor Threshold Get Sensor Threshold Set Sensor Event Enable Get Sensor Event Enable Re arm Sensor Events Get Sensor Event Status Get Sensor Reading Set Sensor Type Get Sensor Type FRU Device Commands Get FRU Inventory Area Info Storage Read FRU Data Storage Write FRU Data SDR Device Commands Get SDR Repository Info Get SDR Repository Allocation Info Reserve SDR Repository Get SDR Add SDR Partial Add SDR Storage Storage Storage Storage Storage Storage Storage Delete SDR Clear SDR Repository Storage Storage Get SDR Repository Time Storage Set SDR Repository Time Storage Enter SDR Repository Update Mode Exit SDR R
34. t 6 1b quiescence acknowledge OS ready 5 1b OS daemon present 4 0 Reserved 2 Quiesce wait timeout sec a An OS daemon is present refer to bits above This is the maximum time from the moment on that the MMC receives FRU Control Quiesce request until it sends back the appropriate Module Hot Swap event message b No OS daemon is present refer to bits above This is the maximum time from the moment on that the MMC receives FRU Control Quiesce request until it sends back the appropriate Module Hot Swap event message If sleep state is recognized before timeout then the Module Hot Swap event message will be sent immediately If the time is set to 0 default after reset then the Module Hot Swap event message will only be sent after recognition of sleep state signal Response data 1 Completion code 00h 2 Control bits 7 Reserved 6 1b quiescence acknowledge OS ready 5 1b OS daemon present 4 1b quiesce request FRU Control 3 Reserved 2 1b graceful reboot request FRU Control 1 1b quiescence reached MMC acknowledge 0 1b module hot swap switch opened 3 Quiesce wait timeout valid only if OS daemon present 1 Settings changed with this command are volatile in particular quiesce timeout and OS daemon present Bits 6 5 are always settable but once the quiesce request comes they cannot be cleared until quiescence
35. tc Page 8 2007 Kontron AG ID 36725 Rev 1 0 AM4100 IPMI Firmware User Guide 6 MMC Firmware 6 1 Key Features compliant with the related IPMI and PICMG specifications refer to 4 2 Related Documentation firmware designed and specially made for AdvancedMC environments supports two KCS interfaces with interrupt support supports the local IPMB IPMB L interface out of band management and monitoring using IPMB L interface permits access to sensors regardless of module CPU state sensor thresholds fully configurable sensor names prefixed with AMC module Bay ID A1 4 B1 4 Usable in uTCA slots 1 12 Sensor names for slots 9 12 are prefixed with R9 R12 complete IPMI watchdog functionality complete FRU functionality refer to 8 4 firmware can be updated in the field refer to 8 2 two firmware banks implemented firmware bank management is done by the open tool ipmitool function fwum 1 refer to 8 1 downloading new firmware image does not break currently running firmware activities refer to 8 2 manual and automatic firmware image roll back in case of upgrade failure Refer to 8 2 firmware customizable per boot loader or per OEM IPMI commands interoperable with other AMC ATCA or IPMI solutions OEM board supervision and control extensions such as boot device flash selection and firmware boot order configuration refer to 6 2 2 Boot loader supervisor and automated switch to recover the bo
36. ttp www picmg org e PICMG AMC 0 R1 0 Advanced Mezzanine Card Base Specification e PICMG AMC 1 R1 0 PCI Express and Advanced Switching on AdvancedMC e PICMG AMC 2 R1 0 AMC Gigabit Ethernet 10 Gigabit XAUI Ethernet Open tools documentation e 1 Ipmitool documentation http ipmitool sourceforge net e OpenIPMI documentation http www openipmi sourceforge net Kontron manuals and specifications http www kontron com e AM4100 User s Guide e 3 AM4100 Linux Board Support Package As a hot swappable field replaceable unit FRU the AM4100 follows the stringent carrier grade RASM feature set namely Reliability Availability Serviceability Maintainability Built in accordance to the AMC 0 specification the AM4100 is also AMC 1 and AMC 2 compliant and is easily managed via IPMI v1 5 v2 0 As with every Advanced Mezzanine Card AMC the AM4100 is equipped with a Module Management Controller MMC ID 36725 Rev 1 0 2007 Kontron AG Page 7 IPMI Firmware User Guide AM4100 r 5 Introduction 5 1 IPMI in AdvancedMC AdvancedTCA Environment The Module Management Controller is a crucial component of any AMC module Besides acting as a regular IPMI management controller sensor monitoring event logging etc it also provides an interface to all necessary data related to module power requirements and implemented interfaces E Keying Further it plays an active role in the module hot swap state mana
37. ufacturers clients literally conserve valuable system AdvancedTCA system slots The AMO010 supports Full Hot Swap capability as per PICMG 3 0 It can be removed from or installed in the system while it is on without powering down the system Please refer to the PICMG 3 0 specification for additional details During Hot Swap of a working module the payload side must be shut down automatically graceful shutdown on command of the MMC and the end of shut down must be signalled back to the MMC Requirements e At system start on payload side the Kontron shutdown daemon grnsd must be started It is included in the Linux board support packages for the AM4100 refer to 3 This daemon communicates cyclically with the MMC for the exchange of states commands and acknowledges For this it uses the OEM Module Quiescence Feedback command Refer to 6 2 2 6 Part of the Hot Swap Operation sequence to be processed by MMC and OS e After the handle switch has been pulled the carrier controller lets the MMC set a shut down request flag for the daemon e The grnsd daemon reads this request flag in the response to its cyclical OEM Module Quiescence Feedback command After having read that flag the grnsd daemon initiates the shut down of the payload software system e At the end of the shut down process the grnsd daemon informs the MMC by setting the appropriate flag when calling the OEM Module Quiescence Feedback comm

Download Pdf Manuals

image

Related Search

Related Contents

CA Service Desk Manager - Manuel d  www.warensortiment.de  FORNAX MOUNTS - LighTrack II -User Manual-  4042526 rev.B SOM  安全上のご注意 サイン資材応援カタログVol.1 ご利用の手引き  M1510/M1515 SoftSkin S40 シリコーンライナー下腿用 カタログ  Windshield Washer Shampoo    

Copyright © All rights reserved.
Failed to retrieve file