C. Paulino, T. Talov, Data Extraction from Solarems Solar Plant
Contents
1. 11 5 Experiments First implementation was done using cUrl library Fig 9 to send a POST request to SolarEMS net with credentials in order to attempt remote login However this approach proved to be unsuccessful and oversimplified because SolarEMS net did not respond as expected Specifically SolarEMS net did not allow remote host to execute POST request and gain access to the system Appendix A Fig 10 cURL request response cycle A second implementation was attempted with using Selenium server and PHPWebDriver This attempt was successful and proof of concept was achieved Fig 10 by developing a basic algorithm that logs in downloads the latest CSV file available onto the local server This affirms that the approach taken will work Also CakePHP s scaffolding options are enabled and a user is able to browse through the database 12 D solar fgcu local E gt DaVinci Navigation Datasets Home Id Title Timestamp Demo Page 1 of 1 showing 0O records out of 0 total starting on record 0 ending on 0 Actions New Dataset development 2 queries took 4 ms Query SELECT Dataset id Dataset title Dataset timestamp FROM solar_app datasets AS Dataset WHERE 1 1 LIMIT 20 SELECT COUNT AS count FROM solar_app datasets AS Dataset WHERE 1 1 Fig 11 Snap shot of DaVinci experimental stage Also basic unit tests were implemented which2. break closing browser session gt close public function download url null row 1 if handle fopen url rb FALSE while data fgetcsv handle 1024 8 FALSE Cache clear num count data if row 1 this gt setFields data yelse this gt setSolarData data if row gt 1 insert array_combine this gt getFields this gt getSolarData exists this gt Dataset gt find list array conditions gt insert if empty exists continue this gt Dataset gt create this gt Dataset gt save insert array validate gt false row Fclose hand1le LER return the fields public function getFields return this gt fields 18 rr param field type fields public function setFields fields foreach fields as value this gt fields Inflector cameLize Inflector slug value this gt fields array _unique this gt fields i public function mysqlFields fields this gt getFields fields array_unique fields foreach fields as value if value local_timestamp value utc_timestamp this gt out ALTER TABLE datasets ADD value DATETIME NOT NULL continue this gt out ALTER TABLE datasets ADD value DOUBLE NOT NULL J aes return the solar
3. http test watercolorgallery eu index php id serveur 2 There are several reasons this type of project may be useful First of all technology is not always reliable so it is good to have redundant backups of critical data The project demonstrates a way to keep a backup of all important data from a website In this case it happens to be data on a solar plant from solarems net If that website were to be hacked or have its server crash the data may be erased or inaccessible when it is needed If there is a backup system set in place data dependent operations may still continue and the website can have its data restored However the main reason to retrieve the data from a remote solar plant server is to do analysis for potential faults of the plant equipment Without constant monitoring of the equipment a fault could go unnoticed and cripple the operation of a solar plant It is important to be able to catch potential faults as soon as possible so the solar plant can continue to run smoothly Previous Accomplishments This project is a continuation of one by C Steiner 1 During that project a client machine named daVinci was created with an objective to connect to the solar plant server and collect data The eBox 4864 was setup and Ubuntu 10 10 was installed It was connected to the FGCU network and all necessary interfaces were established Such interfaces include the graphical interface on an external monitor and I O devices A shared
4. Data ai public function getSolarData return this gt solarData pur param field type solarData say public function setSolarData solarData this gt solarData solarData 19
5. a and presentation MySQL database is used to store all data collected from SolarEMS net service Fig 5 However CakePHP 2 is database independent so this does not impose limitation on how the data are stored or read The database layer can be changed at any time without affecting the integrity of DaVinci 7 SolarEms net e Main Datasource Selenium Standalone Server e Acts like a browser and it browsers solarems net Davinci e DaVinci extracts the data from CSV file and prepares it to be campatibe with CakePHP ORM CakePHP ORM e MySQL query is built at this stage MySQL Database e Data is inserted into the MySQL database Fig 5 Dataflow diagram For data gathering Selenium standalone server 3 is used which provides browser like capabilities This allows DaVinci to browse SolarEMS net as a regular user would which will mitigate the risk of triggering any defence capabilities SolarEMS net might have Fig 6 SolarEMS net Login Page However Selenium does not provide PHP interface which means that PHP cannot work directly with it Thus a mediator is needed that will be able to provide the needed API to Selenium Facebook s PHP WebDriver was chosen as a mediator PHPWebDriver provides basic API for interacting with Selenium but it does provide enough support for this project It supports basic events such as JavaScript s getElementBylIdQ which allows DaVinci to search SolarEMS s Document Objec
6. daVinci Solarems Data Extraction Christian Paulino Teodor Talov Instructor Dr Januz Zalewski CEN 4935 Software Project in Computer Networks Florida Gulf Coast University 10501 FGCU Blvd S Fort Myers FL 33965 6565 Fall 2012 Draft 6 Submission Date November 29 2012 1 Introduction This project dubbed daVincv is about creating a server that is dedicated to downloading data from solarems net which is a FGCU solar plant website The solar plant s data are presented via CSV files CSV stands for Comma Separated Values This type of file stores data in the form of text in a tabular format Records are separated by line breaks and fields are separated by a comma The physical server for the CSV files to be stored is an eBox 4864 Embedded PC 5 The eBox 4864 is a compact embedded PC compatible with Linux operating system Ubuntu 10 10 was the operating system chosen for this project The eBox 4864 runs on a Via Esther 1 2 GHz processor with GB of RAM It can connect to networks via an Ethernet connection and can also interface with other devices through 6 USB 2 0 ports There are PS2 ports for mouse and keyboard connectivity a VGA port for graphical interfaces compact flash port network interface card and wireless adapter All of these components are illustrated in Figure 1 DC Power Jack PST KBs MS LAM hic In Line Gut i Power Switch Compact Flash Slot USS Forts Fig l Back panel of eBox 4864 Source
7. database schema by running the following command php lib Cake Console cake php schema create Answer the prompt with Y Start Selenium in the backgorun by running the following command first navigate to var www and then run java jar selenium selenium server standalone 2 25 0 jar amp Run the following command in order to start importing data from the solar plant php lib Cake Console cake php import 15 References 1 C Steiner daVinci eBox 4864 Sentalis Fetch CSV Server 2011 2 Cake Software Foundation CakePHP the rapid development php framework Pages Cake Software Foundation Online Available http cakephp org Accessed 08 10 2012 3 Selenium Web Browser Automation Selenium Online Available http www seleniumhg org Accessed 08 10 2012 4 S Bergmann The PHP Unit Testing framework Online Available https github com sebastianbergmann phpunit Accessed 03 10 2012 5 Zentyal Inc Zentyal The Linux Small Business Server Zentyal Inc Online Available www zentyal org Accessed 08 10 2012 16 Appendix A lt php App import Vendor php webdriver __init__ class ImportShell extends AppShell public uses array Dataset J public fields public solarData public function main this gt out Import Started this gt _import this gt out Import Complete public function _import se
8. e This allows for easy access to the data as well as an easy way to get valuable information out of it The program does this by navigating to the data page on solarems net and opening the latest CSV file The CSV file contains all the data produced from the solar plant When it comes to a solar plant data can be critical Storing the data in a way that it can be accessed with ease will allow faults to be detected as well as other problems that the data may show With a dedicated server automating the process of data retrieval a lot of potential problems can be dealt with swiftly User Manual The following steps must be taken in order to install and run the software 1 Install any Linux distribution Ubuntu preferred 2 Install Apache 2 2 x 3 Install PHP 5 3 or greater 4 Install MySQL 5 5 or greater 5 Install Java 6 Enable support for PHP and MySQL in Apache 7 Install GIT 8 Install Firefox 9 Navigate to the following directory 14 10 11 12 13 14 var www Clone the following public repository git bitbucket org solarfgcu solar2 git by using the following command git clone git bitbucket org solarftgcu solar2 git BitBucket account with public key required Please create your account at www bitbucket org and add a public key to your account in order to authenticate Create a database named solar_app By executing the following command database create solar_app Install application specific
9. folder was created on daVinci to store the CSV files A screenshot of the page on the solarems net website where the CSV files are downloaded from is shown in Figure 2 In order to retrieve the CSV files a Java program was created This program uses a framework called HTMLUnit It simulates a browser in this case Firefox and navigates through the solarems net website to access the CSV files The program was installed on the eBox 4864 to automate retrieving and storing the CSV files Once installed the Java program only needs to be executed once It runs every hour and downloads all new CSV files since the last time it ran Haimi Vettes PGR ANT ii a Commune at Fyre Tiu Ca aT a i 1AA ae ee aia ee Thea a eee on E ees De les anj 2 ee eas Ct ae et f Mii SS ap S ee ca ee Fig 2 Sample Solarems net data page 3 Problem Description The current implementation of daVinci no longer works The Java program written to retrieve the CSV files can no longer do so The solarems net website changed the way it links to each CSV file Errors of attempts to run the program are shown in Figure 3 The Java program that uses HTMLUnit does not compensate for the new signature added to each CSV file link Besides no longer working it wasn t very efficient to begin with HTMLUnit does not support CSS or JavaScript so it would produce errors when retrieving the CSV files All daVinci did before it stopped working was store the CSV files I
10. provide the following assertions 1 Selenium is up and running This is done by calling the following URL via cURL http localhost 4444 selenium server driver cmd testComplete The expected response from Selenium is OK If any other response is received the test will fail 2 DaVinci can login to SolarEMS net credentials are valid as well as no change as been made to the DOM Document Object Model of SolarEMS net This is done after asserting that Selenium is working Unit test 1 above It is achieved by matching previously saved login form signature to the one that Seleium obtains on run time If they match there has been no change to the Solar EMS s DOM and DaVinci can login screenshots will be included in future draft A database layer has been added with one table which completes the Proof of Concept by combining business logic obtaining the latest CSV file database layer and presentation Solar EMS s user interface was changed which lead to the re implementation of major part of DaVinci s shell scripts that are used to communicate with Selenium 6 Conclusion The old daVinci program quit working so a new daVinci program was written The new program was written in php using the cake php framework Php webdriver from facebook and selenium were also used The new program does more than just create a server for storing redundant data It now puts all data into a database which can be accessed from a web pag
11. t java 354 at SentalisFetchCS fetchCS SentalisFetchCSv java 86 at SentalisFetchCSV nain SentalisFetchCS java 33 davinci davinci _ Fig 3 Old daVinci errors In summary the following steps are proposed for new implementations e Use new technologies Linux Apache MySQL and PHP LAMP e Store CSV files on eBox 4864 e Add CSV file data to a database 4 Solution and implementation Davinci User Interface DaVinci Backend Selenium Standalone Server Ubuntu with LAMP stack Fig 4 Technology Stack Due to incompleteness prior implementation it was decided that DaVinci will be re implemented Respective web technologies were chosen in order to provide internet based interface for users to interact with the system Linux Apache MySQL and PHP LAMP was selected as implementation stack Fig 4 Additionally CakePHP 2 will be used as implementation framework due to its Rapid Application Development RAD capabilities Specifically CakePHP 2 provides out of the box scaffolding features which allow building basic User Interface UI so users can interact with data that are recorded by using CakePHP s 2 built in Create Read Update Delete CRUD methods Additionally CakePHP 2 provides Model View Controller MVC architecture which allows the project to be easily extended in the future Fig 4 MVC architecture also provides clear separation of concerns business logic is separate from dat
12. t Model DOM and determine its state This enables DaVinci to login and browse the website Fig 7 DaVinci will take advantage of the cron tables provided by Linux to execute itself every hour and update the database with the latest data available This is done by taking advantage of CakePHP s 2 shell scripting capabilities Taaa lt M ESYo Vastantsmaleas Browser Fig 7 Physical Diagram DaVinci development follows incremental development process Fig 8 e Installing prerequisites e Ubuntu e PHP e MySQL e Apache e phpUnit e Selenium e Developing modules e DaVinci API to PHP Web Driver e DaVinci shell script 10 e DaVinci database layer e DaVinci scaffolding Ubuntu installed with LAMP Stack Selenium Installed PHP Web Driver Implemented DaVinci Implemented DaVinci API implemented in order to communicate with PHP Web Driver Fig 8 Program Development Steps Testing of DaVinci is done via PHPUnitTest Framework 4 This allows quickly pin pointing possible technical issues as well as asserting that DaVinci is operational In order to run the DaVinci test suite the following command is used in a terminal window Fig 9 php lib Cake Console cake php testsuite app all eodor teodor VirtualBox var www share solar php lib Cake Console cake php testsuite app akePHP Test Shell App Test Cases 1 Controller ApiController 2 Controller DatasetsController 3 Model Dataset CEA
13. t did not provide a way to use the data in a useful way Because of these shortcomings daVinci has to be re implemented A different technology has been chosen to perform the CSV file retrieval process In order for the data to be used practically and not just stored a database has to be used The database will allow for the data to be manipulated and used beyond what a physical disk can provide Drakerlabs who host slarems net do not provide an API which causes implementation to be more difficult 998u 5B5D 31108 0795 cdib DiZi ec ESD 304 38x5 ESB 5142405B5P 3021 v 55 50 30484v58 53 39384v SBSD 30478 075B5B 30398 v7 5B 5D 30408 v SB7SD 304 18 0 5 BSD 36378 e SBSB S 3S apii ae O a 5p 30178075B 5D 30188075B75 30 0 5B 5D 30278075B 5D 3626407SE58 30718075B75D 30658075B75D 30688 v75B75D 30638 v 5B75D 30648 vZSB7SD 5 1648 0 5 B SD 3 0668075B75D 30708 075B75D 305 94 v5 B75 366 98 v 5B75D 36664 95875 366 14 075B753 366 28e75B753 30588 v5 8750 99764 v 5B75D 59808 075B75D 597 78 0758750 36564 v5 8753 38538 675B75B 30498075B750 305 14 075875D 38528 v 5B75D 36558 v5 8753 38566 0 5B750 305 7407 58758 3054 at con gargoylesoftuare htnalunit UebClient throwFailinglttpStatusCodeExc eptionIfNecessary WebClient java 531 at con gargoylesoftuare htnlunit WebClient getPage UebClient java 388 at con gargoylesof tuare htnlunit VebClient getPage UebClient java 369 at con gargoylesoftware htnlunit WebClient getPage WebClien
14. t_time limit startDate date Y m d strtotime now endDate date Y m d strtotime 1 days webdriver new WebDriver session webdriver gt session htmlunit array javascriptEnabled gt true version gt 3 6 session webdriver gt session firefox array session gt open https solarems net session gt element id user _session email gt value array value gt str_split ttalov eagle fgcu edu session gt element id user _session_password gt value array value gt str_split solarfgcu button session gt element id new_user_session button gt submit session gt open https solarems net projects 36 fgcu ab7 data sets 26 exports session gt open https solarems net projects 36 fgcu ab7 data sets 26 exports session gt element id data export _start_date gt value array value gt str_split startDate session gt element id data export stop date gt value array value gt str_split endDate button session gt element id new _data_export button gt submit Ssleep 2 17 source session gt source dom new DOMDocument dom gt loadHTML source primaryContent dom gt getElementBylId primary content with nav foreach primaryContent gt getElementsByTagName a as node this gt download node gt getAttribute href
Download Pdf Manuals
Related Search