SuMegha Cloud Lab Kit
Contents
1. e SFC File No file chosen Select Files With ieee extensions only SIG File No fie chosen Select Files with ieee extensions only Command Line Arguements Submit ij Figure 12 Job Submission page 34 dim COCC job details job is complete check output error file sidebar menu select jobid 35 View Job Status PSE Job Info select output error file Visualization S Help OutputFile Logout ing runscr fcstparm runscr fcstparm runscr fcstparm ken runscr fcstparm ken runscr ftparch runscr ftparch runscr ftpavrg runscr ftpavrg runscr getopini runscr getopini Figure 15 Job Info page 4 Web based File Explorer is provided for the user to view the directories and file and to download the file E psebenchmark4 outdir bak little sig anl 1986042600 ieee J execpsebenchmark44script_jt execpsebenchmark44script_jt e35 b execpsebenchmark4ddscript jt o35 little sfc anl 1986042600 ieee p H little sfc anl 1986042600 ieee bak httle sig anl 1986042600 ieee amp sfc anl 1986042600 ieee sig anl 1986042600 iece 8 E swap x Figure 14 File Browser 5 After completion of job the flx output file is generated This file has to be converted into descriptor file i e ctl file Using this descriptor file the user can do visualization and enter the grads commands in the text area provided2. PaaS Platform as a Service It offers development platform on the cloud Provider manages the cloud infrastructure for the platform and users can develop and host their applications on cloud The entire life cycle of software can be operated on a PaaS ti SoM Example Google App Engine lets users run web applications on Google s infrastructure JaaS Infrastructure as a Service It is on demand provisioning of Virtual Machines Virtual Clusters and Storage StaaS storage as a service to users The main concept being virtualization where the users through their virtual desktop access resources like Network Storage Virtualized Servers Routers and so on supplied by Cloud Service Provider CSP Examples Amazon EC2 and Amazon S3 Storage as a Service Following are the common deployment models of cloud Hyb ri d n in O Fi T T Private FC P i A Internal 1 Public c i Nee k L iN E xt e mi a Fi Y d X a p t The Cloud x On Premises Internal Off Premises Third Party Figure 2 Cloud computing types 1 Public clouds The services and infrastructure in a Public Cloud are provided off site over the Internet that 1s open for public use Usually they are owned by the organization offering the cloud services 2 Private clouds A private cloud is one in which the services and infrastructure are maintained on a private network and is operated solely for one orga
3. 8 8 DNS2 8 8 4 4 DEFROUTE yes IPV4_FAILURE_FATAL yes IPV6INIT no NAME System eth0 UUID 5tb06bd0 0bbO 71fb 45f1 d6edd6513e03 BRIDGE br0 Restart the network re It should now show the network bridge br0 brO Link encap Ethernet HWaddr 00 1E 90 F3 F0 02 inet addr 192 168 0 100 Bcast 192 168 0 255 Mask 255 255 255 0 inet6 addr fe80 21e 90ff fef3 f002 64 Scope Link UP BROADCAST RUNNING MULTICAST MTU 1500 Metric 1 RX packets 17 errors 0 dropped 0 overruns 0 frame 0 TX packets 29 errors 0 dropped 0 overruns 0 carrier 0 collisions 0 txqueuelen 0 RX bytes 1196 1 1 KiB TX bytes 2794 2 7 KiB ethO Link encap Ethernet HWaddr 00 1E 90 F3 F0 02 inet6 addr fe80 21e 90ff fef3 f002 64 Scope Link UP BROADCAST RUNNING MULTICAST MTU 1500 Metric 1 RX packets 4554 errors 0 dropped 0 overruns 0 frame 0 18 WS cocc TX packets 3020 errors 0 dropped 0 overruns 0 carrier 0 collisions 0 txqueuelen 1000 RX bytes 6249612 5 9 MiB TX bytes 254928 248 9 KiB Interrupt 25 Base address 0x6000 lo Link encap Local Loopback inet addr 127 0 0 1 Mask 255 0 0 0 inet6 addr 1 128 Scope Host UP LOOPBACK RUNNING MTU 16436 Metric 1 RX packets 3 errors 0 dropped 0 overruns 0 frame 0 TX packets 3 errors 0 dropped 0 overruns 0 carrier 0 collisions 0 txqueuelen 0 RX bytes 1304 1 2 KiB TX bytes 1304 1 2 KiB Please execute command to switch off Network Manager at boot time Chkconfig NetworkManager off The VMM installatio
4. Resource usage Billing etc WISH CDEC Chapter 2 SuMegha Cloud Stack 2 1 SUMEGHA CLOUD STACK COMPONENTS SuMegha Cloud Stack has been designed keeping in mind the requirements of the scientific environments and applications The stack comprises of the stable versions of best suited Cloud components required to build the private scientific cloud Figure 1 depicts the complete SuMegha software stack The basic SuMegha Cloud lab kit consists of the following components OS CentOS Downloadable from SuMegha website www Sumegha in Hypervisor Xen CD Cloud Middleware Nimbus CD Cloud Portal Pre built golden images with CentOS MPI amp HADOOP environments and tools like job submission portal for virtual HPC clusters VV VV WV SuMegha Cloud Stack Figure 3 SuMegha Cloud Stack Figure 4 shows the architecture of Cloud components installed using SuMegha Cloud Lab kit VMM Node VMM Node Service Node N 5u3 contro e libvi 4 i irt IE Xen KVM Cumulus Clowd client sirope aay V t DHCP Server Figure 4 SuMegha Cloud Architecture 2 2 SUMEGHA LAB KIT OFFERINGS e Basic Cloud lab kit Free Currently available a Software stack to setup Private cloud for provisioning of Virtual Machines Servers i e deployment of Infrastructure as a Service b Interactive installation amp configuration of cloud stack User manual for SuMegha Cloud lab kit D
5. requiretty entry Do not comment the line instead delete the requiretty entry Defaults specification Disable ssh hostname sudo lt cmd gt because it will show the password in clear You have to run ssh t hostname sudo lt cmd gt Defaults requiretty Refuse to run if unable to disable echo on the tty This setting should also be changed in order to be able to use sudo without a tty See requiretty above Defaults visiblepw Providing Sudo privileges to nimbus user In the file etc sudoers please add nimbus user the sudo privileges Add entry nimbus ALL ALL NOPASSWD ALL user MACHINE COMMANDS ES The COMMANDS section may have other options added to it EE Allow root to run any commands anywhere ALL ALL ALL NOPASSWD ALL Allows members of the sys group to run networking software service management apps and more T3 4 ALL NETWORKING SOFTWARE SERVICES STORAGE DELEGATING PROCESSES LOCATE Allows people in group wheel to run all commands twheel ALL ALL ALL Configuration Virtual Machine Manager VMM Node Before proceeding please enable the virtualization in the BIOS setting DRIVERS 11 WISH eEDcc e For Intel processors Go To Advanced Tab gt CPU Configuration gt Intel Virtualization gt Enabled ROM BASED SETUP UTILITY ROM BASED SETUP UTILITY When enabled a UMM can utilize the additional HW Caps p
6. script The installations are done in opt nimbus directory by default Ensure the output of the commands hostname and hostname f should be same Following are the steps for installing Service node 1 Set up password less login using SSH keys to connect nimbus user of VMM and the Service a Generate an SSH keygen keys on the Service Node nimbus service ssh keygen t rsa b Create ssh directory on VMM node nimbus vmm1 ssh nimbus vmm1 mkdir p ssh c Upload the generated public keys to VMM node nimbus service cat ssh id_rsa pub ssh nimbus vmml cat gt gt ssh authorized_keys d Set permissions on the VMM node Due to different SSH versions on servers we need to set permissions on ssh directory and authorized keys file nimbus vmm1 ssh nimbusQ vmml chmod 700 ssh chmod 640 ssh authorized_keys nimbus vmm 1 s password Enter Your Password Here e Login from Service Node to VMM Node without password From now onwards you can log into VMM node as nimbus user from the Service Node 20 m WiSH B Ww as nimbus user without password nimbus Gservice ssh nimbus vmml NOTE As you add more VMMs in future you should ensure that SSH works in the same way for each 2 While installing the Service node the nimbus user 1s created You have to press y to enable it You have to provide the hostname of the machine where you have installed the VMM node If the VMM node is on same machine you c
7. 10 180 36 3 pub4 cdacb in 10 180 36 4 pubS cdacb in 10 180 36 5 pub6 cdacb in 10 180 36 6 pub7 cdacb in 10 180 36 7 Gateway 10 180 36 1 10 180 36 1 10 180 36 1 10 180 36 1 10 180 36 1 Broadcast 10 180 36 255 10 180 36 255 10 180 36 255 10 180 36 255 10 180 36 255 Subnet Mask MAC 255 255 255 0 255 255 255 0 255 255 255 0 255 255 255 0 253253 255 255 0 After adding these entries restart the nimbus services Also add these entries in your centralized DHCP server in the file home nimbus nimbus services var nimbus dhcpd entries Note e All the administration commands are present in the directory home nimbus nimbus bin e Two additional Scripts are provided in home nimbus directory for creating the user and changing password e Cloud Portal is installed in WebApplication Tomcat Server Building the image repository The images are required to boot the VM instances when requested by the user from cloud portal The steps for setting up the image repository are as follows 1 Download the images from the following link http www sumegha in Repository Images 27 CDCC 2 Switch user as nimbus Note All the administrative commands should be executed as nimbus user 3 Run the following command as nimbus user to upload each image to the Cloud image repository home nimbus nimbus cloud client 02 I bin cloud client sh transfer sourcefile lt image path gt common This complete
8. AM Upon clicking the RUN button the Virtual Machine 1s created and listed in the Instance page Following are the three states that Virtual Machine can be in e UNPROPAGATED your machine is in INSLALATION mode e PROPAGATED your machine is in BOOTING mode e RUNNING your machine is ready to use Save the Virtual Machine You can save an instance of Virtual Machine by specifying the Image name and then clicking on Save button in the Instances Images page Note While saving do not destroy the Virtual Machine else you lose both your data and VM Also you cannot save a Cluster Destroy the Virtual Machine Click on the Destroy Machine block in the Instances Images page and select the Virtual Machine handle to be destroyed Note You can destroy the machines at any point of time Once you destroy the Virtual Machine the data in that machine also get destroyed Run the Saved Virtual Machine When you save a Virtual Machine it is listed in the Run Instances list on the Instances Images page Specify the mage Size and the Number of Hours the image is required then click RUN to start the machine using the saved image Delete the saved Virtual Machine Click the Delete button next to the Image name in the list of saved images of the Virtual Machine Note If you delete a saved image it does not delete the Virtual Machine with which it was booted 25 4 3 LOGIN TO VIRTUAL MACHINE 1 From Linux System You can l
9. COCC SuMegha Cloud Lab Kit Version 1 3 User Manual 10 Oct 2013 CDAC System Software Development for HPC C DAC KP Bangalore CDCC SuMegha Cloud Lab Kit Version 1 3 User Manual Project No CDACB SSDH CSC 2013 Document No SSDH SuMegha 2013 LabKit User manual 1 3 Control Status Controlled For External users mentioned in Distribution List Authors Sukeshini Payal Saluja Shivay Veer Sharma Deepanshu Saxena Distribution List Engineering Colleges and Research Organisations Approved By Designation Date of Approval Name Signature Dr B B Prahlada Rao Joint Director SSDH CDAC Bangalore Release By SSDH C DAC Knowledge Park Bangalore Date of Release Copy No 1 Contents Chapter DntPOuuc DO ececsosepe ese ee serer t ovedcsameetecteccasnes woenscescueetectecnosmcusecssome a 4 ELC DOT COMPU srera E EOE E A E EA E NH DIN 4 1 2 SuMegha Cloud Lab kit sare cag Pana RPREFIS accu noes QUE vie ssn a 6 Chapter 2 SuMegha Cloud Stack seeseseeseseccccesesecceseseseoseseseseoseseseseesesesesseecesos 7 2 1 SuMegha Cloud Stack Components ccccssccccccccccssscccccccccsssscccccccccsscees 7 2 2 SUlvies ha Lab Kit OIferimnbs ics oeaoe cesse oe EA XE PESURVIPE SE EENO S EPIFT PENUREEPSESVEP USERS 8 Chapter 3 Installations and Configurations ccccccsccccscsscccccccccccccccccssssssssees 10 3 1 Prerequisites for Installations eeeeceeee eee eee eene ee
10. Cluster nodes have the same name the first one 1s the head node The setup allows creation of two types of Clusters e MPI Cluster e Hadoop Cluster 4 4 1 Creation of MPI Message Passing Interface Cluster In the MPI cluster the machine file root mpd hosts is used for running MPD by using the following command in the Head Node as root user mpdboot n no of nodes f root mpd hosts 4 4 2 Creation of Hadoop Cluster Hadoop cluster allows users to e Run Hadoop MapReduce jobs on the Hadoop cluster e Store data on HDFS Hadoop Distributed File System redundantly ie multiple copies will be stored in the cluster to provide fault tolerance On Hadoop cluster 1 Login to the Master Node and switch to hadoop user 2 Hadoop is installed in home hadoop project hadoop 0 20 0 directory Switch to this directory before performing any operations ie Data storing and Running your Map Reduce JOB 3 Run you Hadoop Mapreduce Job cd Shome hadoop project hadoop 0 20 0 a Copy the local directory to HDFS bin hadoop dfs put localInput home hadoop project hadoop hdfs data dfsInput b List all the HDFS files in the directory to check whether the input directory is present bin hadoop dfs ls home hadoop project hadoop hdfs data c Run your job Ex ABC jar is the file in dfsInput directory 28 cCc bin hadoop jar ABC jar home hadoop project hadoop hdfs data dfsInput home hadoop project hadoop hdfs data df
11. an provide localhost as shown below You have to provide the RAM size allocated for the VMM s which shall be used for virtualization Does the container account nimbus need a special non default ssh key to access the nimbus account on the VMM Node Press n All the VMMs should be able to access the Service Node by hostname The hostname is detected by the installation script and is shown in console If the hostname is incorrect then press no else provide correct hostname By default SSH Server runs on port 22 If the VMM Node s SSH port number is 22 don t change the port number during installation by pressing n If ssh port number is other than 22 then provide the ssh port number Hd server on a port different than 22 21 CDEC 10 For completing the installation you need to explicitly start nimbus services when the script prompts To start the nimbus services login to another terminal as nimbus user and run following command home nimbus nimbus bin nimbusctl start After these services start press Enter 1n installation script 11 If all the entries are correct type y and press Enter when it prompt the message 12 After starting the service in other terminal you should provide the list of IP s required for the Virtual Machines in the file home nimbus nimbus services etc nimbus workspace service network pools public Hostname IP Address pub3 cdacb in
12. as shown in Figure 15 35 sidebar menu Create Deciptor File Wars Select O P file fix t20041986 ctl LoanvenToCH ViewContent Logout Diete d PRESRcE ERE Smet 1 444 mid eicud 53 base Pressure P f IFRE Secs 1 223 mid cleud top Pressuse Fa RESheb 1 232 0 high cleud bese Pressure Pa f PRE3hce 1 233 0 high cloud rcx Pressure Ja DWATcim 54 200 0 atmos column Precipltable water kg m l DNOTsfc 6 0 2 0 murface uncff kg m 2 ISD7RMe c O 222 21 murface Jed pev of tame tend of rel hum 9 pUTUVefc gt 402 21 0 surface Sta Clic To Run Sciel Figure 15 Visualization 36 EDcc LIST OF FIGURES Figure 1 Cloud computing layers Figure 2 Cloud computing types Figure 3 SuMegha Cloud Stack Figure 4 SuMegha Cloud Architecture Figure 5 Puttygen Convert ssh private key to putty compatible format Figure 6 Putty uploading ssh private key to login to the remote machine Figure 7 Login page for Job Submission Portal for Scientific Cloud Figure 8 Job Submission Page Figure 9 Job Info Page Figure 10 Adding users Figure 11 Login page for PSE for SFM Figure 12 Job Submission page Figure 13 Job Info page Figure 14 File Browser Figure 15 Visualization od
13. b by selecting the number of process and can give the related input file having ieee extension Figure 12 shows the Job Submission page of PSE for SFM User can monitor the status of the job by selecting the appropriate job id and after completion of job user can view the output error file s Figure 13 shows the Job Info page 33 PSE For SFM login Logind Password sidebar menu iv ome s About Us 5 Help CDCC pse Seasonal Forecast Model SFM is an efficient stable state of the meene art atmospheric general circulation model designed for seasonal prediction and climate research It was developed by Experimental Climate Prediction Center ECPC USA It is a free resource available to the research and academic communities under research license SFM is a global spectral model with hydrostatic primitive equation on sigma coordinate It uses spherical harmonic function with Fourier series in east west direction and associated Legendre function in North South direction for representaion of fields in horizontal and finite difference representation in vertical Leap frog scheme is used for time integration It is designed using SPMD Single Program Multiple Data programming model It can run on sequential Figure 11 Login page for PSE for SFM PSE For SFM sidebar menu Job Info Visualization Help Logout Job Name Set Type Select Job Type
14. d Lab Kit installation 3 1 1 Virtual Machine Manager VMM Node Operating System Centos version 6 4 Downloadable from link http www sumegha in Repository CentOS_6 iso CPU One or more 64 bit x86 CPU s 1 5 GHz or above 2 GHz or faster multi core CPU recommended Minimum 4 GB Disk Space Minimum 60 GB Minimum 2GB for boot partition 3 1 2 Service Node Disk Space Minimum 100 GB JAVA 1 67 Python 2 6 3 0 3 2 COMMON CONFIGURATIONS Before installations the administrator must check for three things on the both VMM node and Service node 1 Disable the SELINUX 2 Disabling requiretty setting 3 Providing sudo privileges to nimbus user 10 Disable the SELINUX WISH CDEC By default SELINUX is enforcing The user has to disable the SELINUX in the file etc selinux config Please remove enforcing and add the entry SELINU X disabled root scicloud vi etc selinux config This file controls the state of SELinux on the system SELINUX can take one of these three values enforcing SELinux security policy is enforced permissive SELinux prints warnings instead of enforcing disabled o SELinux policy is loaded SELINUX enforcing if SQ INUXTYPE gam take one of these two values targeted Targeted processes are protected mls Multi Level Security protection SELINUXTYPE targeted Disabling requiretty setting In the file etc sudoers discard the
15. erFS and in house developed tools like Cloud portal Problem Solving Environments PSE Cloud Vault Job Submission portal It also provides a set of Golden Images of different sizes like small medium and large with CentOS operating systems and parallel environments with MPI libraries and Hadoop framework This lab kit enables the automated deployment installation and configuration of private cloud on the recommended hardware to setup Cloud Lab without having expertise in system administration Highlights Builds a private cloud in an organization Web based tools that supports easy and quick access to the Virtual machines and Virtual HPC clusters Simple and quick installation of all the cloud components Pre built OS images with complex parallel environments like MPI and Map Reduce to cater the HPC needs Golden images with preloaded HPC applications like Seasonal Forecast Model Cloud Next generation pipeline with the required visualization tools and job submission portal for easy parallel job submission Who can use SuMegha Cloud Lab Kit Entry Level Engineers o To get better understanding of basics necessary functions and features of SuMegha before working on Real time environment FEducational Training Institutes o To setup Cloud laboratory for experimental practical purpose Research Organizations o To identify issues and vulnerabilities of important aspects in Cloud Computing such as Monitoring Security
16. in experts who want to build a private cloud for their applications don t know where and how to start The SuMegha Cloud Lab kit becomes an ideal solution to cater their requirements SuMegha is a C DAC s scientific cloud with various open source and indigenously developed components installed to facilitate the users to execute their applications on cloud with ease Cloud Clients Web browser mobile app thin client terminal emulator 5aa5 CRM Email virtual desktop communication games PaaS Execution runtime database web server development tools Application Plathoren laas Virtual machines servers storage load balancers network infra Structure Figure 1 Cloud computing layers Cloud Computing is delivery of Computing as a Service rather than a product wherein resources software s and information are provided as a utility like the electricity over Internet Cloud computing use the Internet technologies for delivery of IT Enabled capabilities as a service to the required users We can access these resources from anywhere and pay according to our usage The services offered on cloud are SaaS Software as a Service The users are given access to software applications hosted by the cloud vendor over Internet Examples Cloud based word processing application Online e mail providers like Google s Gmail Yahoo mail and Microsoft Hotmail Photo editing softwares Google docs etc
17. indow oe Attempt authentication using Pageant a E Attempt TIS or CryptoCard auth SSH 1 Translation Attempt keyboard nteractive auth S5H 2 2 Selection E E Authentication parameters i Colours Allow agent forwarding E Allow attempted changes of usemame in SSH 2 i Prox Private key file for authentication Telnet Browse SSH z Tunnels L Bugs Figure 6 Putty uploading ssh private key to login to the remote machine 4 4 VIRTUAL CLUSTERS To create a Virtual Cluster provide the following four parameters in the Run Cluster block on the Instances Images page e Type Cluster type e Size RAM and CPU of your Cluster e Node Number of Nodes required including Head Node e Hours Number of hours you want the Virtual Cluster The three variations in Image Sizes are e SmallImage 1vCPU amp 1GB RAM e Medium Image 2vCPU amp 2GB RAM e large 4vCPU amp 4GB RAM Upon clicking the RUN button the Virtual Cluster 1s created and listed in the Instance page Following are the three states that Virtual Machine can be in e UNPROPAGATED your machine is in INSLALATION mode e PROPAGATED your machine is in BOOTING mode e RUNNING your machine is ready to use 21 Note e Do not login to any of the Virtual Cluster nodes o Until all of them are in running mode o Until the Contextualization Status in LOGS CTX SUCCESS is displayed before all the nodes e All the
18. istribution through CD and downloadable from SuMegha website www sumegha in e Basic Cloud lab kit with Cloud Storage facility upto 2 GB Free In future a Software stack to setup Private cloud for provisioning of Virtual Machines Servers 1 e deployment of Infrastructure as a Service IaaS b Interactive installation amp configuration of cloud stack C d e User manual for SuMegha Cloud lab kit Access to Cloudvault cloud storage up to 2 GB free beyond it chargeable per SGB Distribution through CD e Advanced Cloud lab kit Chargeable In future a oOo g Software stack to setup Private cloud for provisioning of Virtual Machines Servers i e deployment of Infrastructure as a Service IaaS Interactive installation amp configuration of cloud stack Software stack to setup cloud storage Storage as a Service StaaS Access to Cloudvault cloud storage Upto 5GB User manual for SuMegha Cloud lab kit CDCC f l day hands on training g Distribution through CD e Support Services v E mail support free v Annual maintenance through E mail support Chargable v One day hands on training chargable COCC Chapter 3 Installation amp Configuration 3 1 PREREQUISITES FOR INSTALLATION It is assumed that the installer possess the knowledge of basic UNIX system administration Following are requirements to set up the Virtual Machine Manager VMM node and Service Node as part of SuMegha Clou
19. ll the options to install VMM node or Service Node shall be displayed 13 rameez rameez VirtualBox Downloads SUMEGHA sumegha Nimbus xen SuMegha Install 3c e e e e e eee e e e eee ee eee xA Xx Welcome to Sumegha installation ERKKKKKKKKKKKKKE Choose the type of Installation Note VMM node should be installed amp configured first followed by the Service node 3 3 1 Virtual Machine Manager VMM Node Installation When the pre requisites are met and VMM node installation option selected the first component of SuMegha Cloud Lab kit Xen installation starts The user is prompted whether to create a default Network Bridge xenbrO or not At any point when the installation has paused please press Enter key twice to continue If user opts for default Xen bridge creation then he she has to provide IP Address Gateway Address DNS Address and the name of the Interface on which network is configured 14 installing leben Role auo M E ves T ia Do you want to i ze Default Xen Bridge Please confirm Enter the DN5 Server The values Enters are In xen bridge function Th Enter the Interface name ethl Shutting down interface ethl bridge zenbr0 does not exist l Shutting down lx k interfac Bringing up loopback interface Bringing up int ethl Bringing up i Network Service Start aoe Hh luec eexes a After bridge gets created installation of Xen its relative packages and libvir
20. n is complete Please reboot your system with Xen kernel and check whether installation is correct using following command virsh list root sumegha virsh list 0 Domain 0 running The following output implies the system has booted with Xen kernel Also check using uname r Please uncomment following lines in etc libvirt libvirtd conf file e Uncomment the line unix sock group libvirt and replace ibvirt with root unix sock group root e Uncomment the following lines unix sock ro perms 0770 unix sock rw perms 0770 unix sock dir var run ibvirt auth unix ro none auth unix rw none e After editing this file run the following command usermod G root nimbus e Restart libvirt and Xen service from root login service libvirtd restart service xend restart 19 CDCC Checking Xen and libvirt If VMM and Service Node are installed on the same machine execute the following command to check whether installations are correct virsh c xen ssh nimbus service node IP list If VMM and Service Nodes are installed on different machines execute the following command to check whether installations are correct virsh c xen ssh nimbus vmm node IP list If you encounter any errors whether during installations or configurations please report to SuMegha team 3 3 2 Service Node Installation Select the Service Node installation option in the main
21. ne eee eoe nnno ne 10 2 2 Common CODIIOUE ALIONS cases roba eF RUE E I EaDDEHEPIS M pIMBUE M INI DL ELERLIM UNDE 10 3 3 Cloud Lab Kit Software Installations cccccccccccccccccccccccccccccccccccccecess 13 3 3 1 Virtual Machine Manager VMM Node Installation 14 3 3 2 Service Node Installation souectusxe oak 9 E ESPqE TES RES ENDE S ISSUES IIIS DIS PSU Oe S OP OPEEE 20 Chapter 4 Using Cloud Services 2255 0disoe ciudad a Estox ERU RE ENDOUS OS IPIM TIU SN PUN I e POLI eMe SUE 24 Aol SII IU T i E IEEE 24 4 2 N CU AN NO IMINO oes oodosque ipd asud eau ois nod seas E oe dd dusque qd P NIIS ORIUNDUS 24 4 3 Login fto Virtual Machines oeste tesi op eERaPpAD ROI a IINE OU EDI GE E PUFEDI P OII DN UTI aaas 26 qd WEN CTUSDOES o6 1oamecTa veers E Pad dan sd E Pe OIL MULPIIDRIUDLUPEIUI EnPDDE MEE 27 4 4 1 Creation of Message Passing Interface MPI Cluster 28 4 4 2 Creation of Hadoop Cluster c eee oper sirE v ER ERE UROPEF PES EUEEO ES PERSA TEF OPES ESAE 28 Clapler e cinipaviri4rzi e v C 30 S T ODSSUDIDISSIOIEPOEUILasuanoanuuiuyadauoddealiad uid usguuE lada dU cara liud uid nada cei qdd D oeU DIE qud MdGU 30 5 2 Problem Solving Environment PSE for SFM cce eeee eere eene 33 List of Figures COCC Chapter 1 Introduction 1 1 CLOUD COMPUTING Cloud Computing is the buzz word in today s technological era The researchers academicians and doma
22. nization These clouds offer the greatest level of security and control 3 Hybrid clouds A hybrid cloud is a composition of two or more clouds private or public and includes a variety of public and private options with multiple providers Scientific Clouds Scientific computing deals with solving large scale scientific problems in domains like astrophysics mechanical engineering and material science by utilizing mathematical modeling and computer simulations Running large and accurate simulations requires a significant amount of computing resources often demanding the utilization of supercomputers Clusters or Grids which may not be available for the researchers and scientists Scientific Clouds can meet these huge computation capabilities and storage requirements of scientific community through 5 cacc infrastructure and storage services that can be shared with the scientists and researchers over Cloud Amazon Elastic MapReduce is an example that gives processing capability for vast amounts of data stored over amazon S3 service SuMegha is a scientific cloud developed by C DAC offers High Performance Compuitng HPC as a service Infrastructure as a Service and Storage as a Service 1 2 SUMEGHA CLOUD LAB KIT SuMegha Cloud lab kit is a sophisticated Cloud installation package comprising of various softwares to build private clouds The Cloud software stack consists of open source components like Xen Nimbus Openstack swift Glust
23. ogin to the Virtual Machine using the SSH key given at the home page by following these steps e Copy the SSH key content into a file e Change the mode of that file using the following command chmod 0600 filename e Now ssh using that key ssh i path of ssh key file root machine IP address 2 From Windows System Step 1 Install putty client on your Windows desktop machine Step 2 Copy the ssh key from the Cloud Portal after registration and save text file on your Windows desktop machine Step 3 Install puttygen Using puttygen as shown in Figure 5 convert ssh private key to PPK format compatible for putty Click yes when prompted for passphrase free ssh communication Pe i ne m um Key Conversions Help Load private key Save public key Save private key Exit Actions Generate a public private key pair Load an existing private key file Save the generated key Parameters Type of key to generate SSH 1 RSA SSH 2 RSA 5 SSH 2 DSA Number of bits in a generated key 1024 Figure 5 Puttygen Convert ssh private key to putty compatible format Step 4 Provide the IP address and Port number of the VM instance to be accessed Step 5 Load the ssh private key as shown in figure 6 to open the session Step 6 Login as root user 26 Er Terminal Options controlling SSH authentication i Keyboard Bell E Bypass authentication entirely S5H 2 only Features Authentication methods E W
24. rovi ded by Intel D Uirtual ization Tech Note fi full reset is requried to change m the setting Options Disabled Enabled C1E Support Select Screen Harduare Prefetcher Enabled Enabled Intel Uirtualization Tech Enabled Core Multi Processing Enabled Intel SpeedStep Tech Select Item Change Option General Help Adjacent Cache Line Prefetch F10 Save and Exit E nab l ad i ESC T LN LAIL v02 61 C Copyright 1985 2006 American Megatrends Inc e For AMD processor Go to System options gt Processor Options gt Amd Virtualization gt Enabled 12 arial Port Options mMnenier ICs 3 3 CLOUD LAB KIT SOFTWARE INSTALLATION SuMegha installation is completely automated with minimal interactions with the user installing the software Also ensure if all the pre requisites are met before installation The installer is available in CD or can be downloaded from SuMegha website http www sumegha in Repository SuMeghaLabKit iso If the installation is done from CD execute the SuMegha Install script present in the main CD directory using the command sh SuMegha Install If the installation is done from downloaded 1so follow these steps mount o loop SuMeghaLabKit iso mount directory b cd mount directory c SuMegha installation is invoked by executing the following script present in main the CD directory sh SuMegha Install When you invoke the main script SuMegha Insta
25. s the installation of Service Node For any issues errors regarding the SuMegha Cloud Lab Kit installation please contact the SuMegha team for clarifications 23 COCC Chapter 4 Using Cloud Services 4 1 CONFIGURATIONS Before starting the cloud services you must ensure the following 1 In Virtual Machine Manager VMM Node e Ensure the system is booted from the Xen Kernel e Check xend service is running 2 In Service Node e Providing IP addresses to Virtual Machines VM and centralized DHCP Server e Nimbus services are running 3 Configuration of Cloud Portal Following parameters in the file WebApplication webapps ROOT WEB INF classes config properties 1s modified as root e Ipaddress x x x x e adminld abc e adminPasswd abc If you face any issues while creating VMs through portal please look into the file WebApplication logs catalina out or contact SuMegha team 4 2 VIRTUAL MACHINES Refer Online help available 1n Cloud Portal for further details Create a Virtual Machine To create a Virtual Machine provide the following three parameters in the Run Instance block on the Instance or Image page e Image File VM Type provided to you or VM s saved by you e Size RAM and CPU of your machine e Hours Number of hours you want the Virtual machine 24 CDCC The three variations in Image Sizes are e Small Image 1 vCPU amp 1GB RAM e Medium Image 2vCPU amp 2GB RAM e large 4vCPU amp 4GB R
26. sOutput Note Output is the name of the directory where the results of Map Reduce job will be stored Ensure this directory does not exist before running the job ABC jar is the job that has to be transferred from your machine to Master Node of Hadoop Cluster Hadoop Programming can be done using Hadoop plugins available for Netbeans Eclipse 4 Copy these output files in dfsOutput directory to your local machine in the directory local Output 29 COCC Chapter 5 Getting Started 5 1 JOB SUBMISSION PORTAL 1 User can access the Job Submission Portal by typing the following URL in the browser http Ip Address 8084 JSPC LOGIN ABOUT US HELP Job Submission Portal for Scientific Cloud Updates And News Access Grid Solution from CDAC GARUDA Password UserName Figure 7 Login page for Job Submission Portal for Scientific Cloud 2 User can submit the Sequential and Parallel Jobs by selecting the appropriate option from the Job Submission Page For parallel jobs the user can select the number of process 3 User has to provide the executables along with the following parameters like input files if any stdin input file command line arguments 30 ot ee poi Job Submission Portal for Scientific Cloud Job Name Job Type Sequential Executable Batch File Remote mpd hosts input File Select File Select Remote File STDIN Select Select Remote File Command Line Arguemen
27. t Figure 8 Job Submission Page 4 After submitting the job the user is given a Job ID for checking the status of job in JobInfo page 5 User can view download the output error file s as shown in Figure 9 6 The administrator can add a user using Add User facility in Figure 10 3l CDCC v Ls EN e Job Submission Portal m for l Scientific Cloud ae Jobid Name User Time Use Status Queue Job Status 103 pub31 ostssamplejt cdac 00 00 00 C bath Fy Running on host ime is Tue Sep 18 13 03 17 IST 2012 Directory is home cdac Job Submission Portal for Scientific Cloud Add Users UserName Password Retype Password l Add User Logout Loain As Non Root Login AS NOn AOO Figure 10 Adding users 32 WS CDCC 5 2 PSE S FOR SFM The Problem Solving Environment PSE for Seasonal Forecast Model SFM is a web interface to interact with the Virtual Clusters provided by scientific cloud to run the SFM application for weather prediction Hardware Requirements Cluster of MPICH2 with Linux OS A cluster of MPICH2 having Torque as a scheduler Software Requirements Apache tomcat Web Application server for communication Mysql Database to store information Torque Scheduler for Linux clusters GrADS for Visualization 1 User can access the PSE for SFM Portal by typing the following URL in the browser http Ip Address 8080 pse sfm User can submit the weather forecasting jo
28. t begins 15 Q Q Q Q 4 2 64bit for pac 3 0 64bit for pa g 4 2 64bit for package x iu ig After the Xen is installed successfully check the default variable in the file boot grub menu lst before rebooting Change the default kernel to Xen Kernel by changing the value of the default variable Creating your own Network Bridge If the user opts for creating his her own Network Bridge he she has to set up a network bridge on their server for virtual machines to be accessed from other hosts as if they were 16 physical systems in the network To do this we install the package bridge utils and configure a bridge Create the file etc sysconfig network scripts ifcf g brO Please use the IPADDR PREFIX GATEWAY DNS1 and DNS2 values from the file letc sysconfig network scripts ifcfg ethOfile Also ensure you J use TYPE Bridge not TYPE Ethernet DEVICE br0 NM_CONTROLLED yes ONBOOT yes TY PE Bridge BOOTPROTO none IPADDR 192 168 0 100 PREFIX 24 GATEWAY 192 168 0 1 DNS 1 8 8 8 8 DNS2 8 8 4 4 DEFROUTE yes IPV4_FAILURE_FATAL yes IPV6INIT no Modify etc sysconfig network scripts ifcf g ethO as follows Comment BOOTPROTO IPADDR PREFIX GATEWAY DNSI and DNS2 and add BRIDGE br0 DEVICE eth0 NM_CONTROLLED yes 17 COCC ONBOOT yes HWADDR 00 1 E 90 F3 FO 02 TY PE Ethernet BOOTPROTO none IPADDR 192 168 0 100 PREFIX 24 GATEWAY 192 168 0 1 DNS 1 8 8
Download Pdf Manuals
Related Search