Russian Flu Project - VTechWorks
Contents
1. Drop raster files with maps here Open a file KLOKAN TECHNOLOGIES Figure 23 Select the 1889 Russian Flu map image to overlay Mapliler FREE SRESER o ANN RICO ee XX S T lt A z Al rh SS S SS U bekistar yrjyzst n2 C DT NS Ss S Y d TA a Kare etr gt Kat Z f A uL n1 ores 4 E i gzhou Pata ar chi Taipei EA Masqat Ahr L aan E Rek nali ong Sette oe Mumba j Kong y A AE T Perd k 2 l T Vietnam nila i hennai Bangkoli Certess Philippines o Chi Colomiba i sri Penh Minh City sei Lanka ON Brunei GALA ala Lumpur e m 8 ala Sia n T A Ja Bandur as Surabaja Timor Lesl KLOKAN TECHNOLOGIES Figure 24 1889 Russian Flu Map image and current Google Maps compared This YouTube video shows how to overlay and image over a map https www youtube com watch v eJxdCe9CNYg To download MapTiler see some of the features and tutorials http www maptiler com The final result is displayed in Figure 25 See Figure 1 for the file location of the map Image IBCAO Data SIO NOAA U S Navy NGA GEBCO Google earth Image Landsat Figure 25 Final Result from MapTiler Application with 1889 Map overlayed 7 2 Timeline For the creation of the timeline we used the open source tool TimelineJS created by Knight Lab TimelineJS allows you to easily create any timeline using a Google Doc spreadsheet containing2. search full text search box and spell checker The facets defined for the Russian Flu project included Newspaper Title Infection Location Reporting Location Language Date and Keywords Once a facet is specified a search is executed or a combination of the two a list of corresponding results is displayed The metadata associated with each article can be viewed as a single document through clicking the Newspaper Title link The website also includes multimedia resources tracking the Russian Flu pandemic An interactive timeline depicts a world map with the flu outbreak and spread at different time intervals Another multimedia resource is the Google Earth 1889 Russian Flu map overlay This interactive Google Map allows a user to see all continents and the flus impact through an 1889 world map The Russian Flu website also has a user log in search history and article bookmark feature A user can bookmark an article upon viewing it as a single document When logged in a user can view a full list of bookmarked articles The search history feature saves a list of the user s recent searches and at the click of a button the search executes again Various problems were encountered throughout the semester the largest being the lack of a server until April This kept us from integrating Solr and Blacklight until access to a server was granted Communication lacked amongst the CS and translation teams in that the CS team relied on the transla
3. type string indexed true stored false multiValued true dynamicField name _facet type string indexed true stored false multiValued false lt copyField copyField copyField copyField copyField copyField copyField copyField copyField copyField copyField source Newspaper Title dest collector source Infection Location dest collector source Infect Lat Long dest collector gt source Reporting Location dest collector source Report Lat Long dest collector source Date dest collector source Link dest collector gt source Translation dest collector source Language dest collector gt source Tags_Keywords dest collector gt source Image Link dest collector gt lt fields gt copyField source _facet dest collector lt uniqueKey gt id lt uniqueKey gt lt defaultSearchField gt collector lt defaultSearchField gt solrQueryParser defaultOperator O0R gt lt types gt fieldType name string class solr StrField sortMissingLast true gt fieldType names text general class solr TextField sortMissingLast true fieldType names text general class solr TextField positionIncrementGap 100 gt lt analyzer type query gt tokenizer class solr StandardTokenizerFactory gt lt filter class solr LowerCaseFilterFactory gt filter class solr WordDelimiterFilterFactory preserveOriginal
4. The Times M In Bookmarks Newspaper Title The Times Infection Location St Petersburg Reporting Location London Date 1889 11 25T00 00 00Z Translation Russia St Petersburg Nov 23 During the week there has been a most unusual case of sickness among all classes in St Petersburg which from the semblance of the symptoms is supposed to be an epidemic of influenza It is averred that scarcely a family has escaped while the numbers of those attacked by it in factories workshops and schools are quite phenomenal In one of the larger mills as many as one hundred workmen at one time have been ill and parts of the work have had to be suspended The hospitals are overcrowded and the doctors complain of too much to do Most of the journals take it as a text for lamentations over the proverbial insalubrity of the city others attribute it to the horrible weather of the last few weeks which has consisted of a series of rapid changes between frost and heat snow and rain with heavy clouds and semi darkness But if St Petersburg is the most You can view your search history by clicking on the History tab This is what the page looks like All Fields v Search Search History Your recent searches Kevword sick Clicking the login button allows a user to create an account Developers Manual J9AJ9S n ueissny uo JINJINIS JIU ou JO WeASEIP 3911 TIS 9 ASV ILVQ Bep nj sme me e e que JWQU JUDIJHOBIQ Q
5. blacklight service blacklight jetty start 20 6 3 Indexing Data As discussed in the Section 3 2 Apache Solr Common Tasks the data to be searched and viewed on the website must be indexed in Solr Below are the steps to upload data 1 2 Download the translation excel spreadsheet on Google Drive as a tab separated values tsv file Secure copy the tsv file from your computer to the server user scp FILENAME YOURPIDGrussianflu lib vt edu SSHinto your Russian Flu directory user ssh YOURPID russianflu lib vt edu Copy the tsv file to the solr directory blacklight home user cp FILENAME blacklight blacklight jetty solr Go to the solr directory blacklight home user cd blacklight blacklight jetty solr Execute the following command to upload the data blacklight blacklight blacklight jetty solr curl http localhost 8983 solr update csv7commit true s eparator lt 09 amp stream file solr FILENAME csv amp stream co ntentTvpe application csv charset utf 8 6 4 Clearing Deleting Data As discussed in Section 3 2 Apache Solr Common Tasks the data to be searched and viewed on the website must be altered in Solr If the schema xml or solrconfig xml file is changed the indexed data must be dropped and re indexed for changes to appear Below are the steps to drop data 1 21 Execute the following commands blacklight curl http localhost 8983
6. solr update H Content type text xml data binary lt delLete gt lt query gt x x lt query gt lt delete gt blacklight curl http localhost 8983 solr update H Content type text xml data binary commit 2 Now follow the instructions in Section 6 3 above to re index the data 7 Multimedia 7 1 Map The map that shows the overall Russian Flu spread was made using the MapTiler software Figure 22 This software allows the use of an image to be overlaid on top of a Google Earth globe MapTiler must be downloaded onto a local machine We needed to use this software to allow us to correctly represent country borders since we were working with data and images from a different decade We were provided with an 1889 map to overlay on top of the globe Using Photoshop this map was altered to only contain the important pieces of the image The software was then used to upload the image and recognize certain parts on the map to be able to overlay the image correctly Figure 24 eoo gu MapTiler Fast maps for web and mobile WE 0 Ei zd FREE Google Earth KML SuperOverlay Custom Tiles Local coordinate systems Mercator Tiles Google maps Web amp Mobile apps GIS Retina HiDPI Tiles mq s Raster Tiles Not georeferenced images KLOKAN U TECHNOLOGIES Go back Continue Figure 22 MapTiler Application used to create 1889 Russian Flu Overlay 22 Mapliler FREE 4 Ye m m
7. 00 articles at a time it was often impossible to locate the errors in the indexable file Solr threw errors regarding incorrect formatting but they were difficult to decipher These persistent errors lead to us removing a few articles from the upload altogether Discussions regarding formatting were held at weekly meetings but the importance of formatting was difficult to project A similar and frequent issue occurred with Solr s formatting of dates The expected format was YYYY MM DDTHH MM SSZ which was contrary to what was currently stored We informed the project team to change the article dates to the correct format but it took longer than expected to accomplish this task and in the meantime we used demo data Once we indexed article data we encountered search configuration issues Full text searches only returned case sensitive exact matches For example when searching for articles written in Spanish spanish did not return any results but Spanish returned all articles with Spanish as their original text We decided to add Solr Factories and Tokenizers to the schema xml field types to combat this issue The lowerCaseFilterFactory parsed all data as lowercase letters which allowed case insensitive results The final problem encountered involved a broken link from the search results page to a single document view This problem has not yet been solved but there is hope to have a fix by the end of the semester Additional
8. 1 gt analyzer fieldType fieldType names date class solr TrieDateField precisionStep 0 positionIncrementGap 0 gt fieldType names long class solr TrieLongField precisionStep 0 positionIncrementGap 0 gt fieldType names text en class solr TextField positionIncrementGap 100 gt lots of details gt lt fieldType gt fieldType names currency class solr CurrencvField precisionStep 8 defaultCurrency USD currencyConfig currency xml lt types gt lt schema gt Figure 2 Schema xml file for Russian Flu website 11 3 4 Solrconfig xml Solr s search capabilities are defined within the solrconfig xml file Figures 3 6 Currently there are two types of searches defined for the Russian Flu website Default search and Document search The default search defined as search in solrconfig xml represents all capabilities on the homepage including default search result fields Figure 3 and facet fields Figure 4 Document search is use to display a single record Figure 5 str name fl gt name facet true str Newspaper Title name facet mincount 1 str Infection Location str name facet limit 18 str Reporting Location name facet field Newspaper Title str Date name facet field Infection Locationt str Link name facet field Reporting Location str Language name facet field Language str Tags Keywords name facet field D
9. Blog Multimedia Figure 10 Final Russian Flu homepage All Fields sal infected Limit vour search You searched for infected X Start Over Newspaper Title Previous 1 10 of 16 Next gt 10 per page Infection Location Reporting Location 1 Intelligenzblatt und Berner Stadtblatt Newspaper Title Intelligenzblatt und Berner Stadtblatt Infection Location London Language Reporting Location Bern Date 1890 01 06T00 00 00Z Language German Keywords postmaster Salisbury 2 La Lanterne Newspaper Title La Lanterne Infection Location Paris Date 1890 01 03T00 00 00Z 3 Intelligenzblatt und Berner Stadtblatt Newspaper Title Intelligenzblatt und Berner Stadtblatt Infection Location London Madrid Brussels Paris Figure 11 Final Russian Flu search results page A few other examples of websites using Blacklight like AGNIC ALICE Columbia University etc can be found here https github com projectblacklight blacklight wiki Examples 16 5 Integrating Solr and Blacklight 5 1 Facet Search Using the field names defined in schema xml the following images depict the implementation of faceted search add facet field Newspaper Title label gt Newspaper Title add facet field Infection Location label gt Infection Location add facet field Reporting Location label gt Reporting Location add facet field Language label gt Language limit gt t
10. J8 JWJUIXA euou CB gt MA C 1uBipyoe q 9 1 Introduction The Russian Flu website is the product of the integration of Apache Solr and Blacklight a Ruby on Rails User Interface This site allows a allow a user to search indexed historical articles documenting the Russian Flu pandemic and view the corresponding search results and multimedia This manual addresses the set up associated with russianflu lib vt edu as well as the necessary steps to maintain it 2 Getting Started 2 1 Requirements In order to use and edit russianflu lib vt edu one must have a stable Internet connection and access to the computer s terminal The first thing needed is an installation of Ruby and Rails for Blacklight and Java for Solr The current Solr release requires Java 1 7 or greater and the current Blacklight release requires Ruby 1 9 or greater Rails 3 2 and 4 x although Rails 4 is recommended and Java 1 6 or greater 2 2 Logging In A user must have an account on the russianflu lib vt edu server in order to access any files associated with the Russian Flu website Once access is granted a user can ssh to the Russian flu server by running the command ssh YOURPID russianflu lib vt edu in a terminal The Russian Flu server can also be accessed through rlogin Once on the server a user will find themselves within their home folder at the full path home YOURPID The Blacklight and Solr files are lo
11. Russian Flu Project Group Members Samantha Fisher Josh Horn Johanna Pinsirikul Taylor Sims Client Tom Ewing CS 4624 Virginia Tech Blacksburg VA opring 2014 Table of Contents Executive Summary User s Manual Searching Links and Other Features Developer s Manual Introduction Getting Started Apache Solr Blacklight Integrating Solr and Blacklight Site Maintenance Multimedia Lessons Learned Acknowledgements References 10 13 17 20 22 28 30 31 Executive Summary This report includes a detailed description of Dr Ewing ss Russian Flu project This project integrated two groups a Translation team and a CS team The translation team worked to find and translate historical articles documenting the Russian Flu pandemic from 1889 to 1890 The languages consisted of French German Spanish English and Russian The CS team indexed the article metadata into a searchable website The desired website allowed a user to search for articles view a list of corresponding results and understand the specifics of a given article The website was implemented using Solr an open source search engine platform and Blacklight a Ruby on Rails gem Solr s search features included facets relevance definitions spell checking and synonyms Blacklight was easily integrated with Solr in that it displayed the search features and results in a user friendly format Blacklight was configured with a faceted search drop down menu
12. abase Without their hard work we would not be able to display all the useful information for the researchers that will be using the site we have created Beth Lancaster for setting up the server for us and helping us with its use Finally a special thanks to our Professor Dr Fox for providing us a path to go with in our project his helpful suggestions in resources we were not familiar with and connecting us with the right people to get our server in place 30 References Apache Lucene Apache Lucene The Apache Software Foundation 2012 Web March 2014 lt http lucene apache org solr tutorial html gt Beer Chris Projectblacklight blacklight GitHub Project Blacklight Apr 2014 Web Mar 2014 lt https github com projectblacklight blacklight wiki Quickstart gt CSS Tutorial CSS Tutorial W3Schools n d Web 18 Apr 2014 lt http www w3schools com css default asp gt Fast Maps for the Web and Mobile MapTiler Klokan Technologies GmbH n d Web Mar 2014 lt http www maptiler com gt Fox E A Multimedia Hypertext and Information Access Professor personal communication May 2014 Lancaster B Sr UNIX Systems Database Administrator personal communication Apr 2014 StoryMapjJS StoryMap JS Northwestern University Knight Lab n d Web May 2014 lt http storymap knightlab com gt TimelineJS Timeline JS Northwestern University Knight Lab n d Web Apr 2014 lt http
13. acklight blacklight app assets images directory Figure 1 To reference the image in one of the html erb files use it like lt image_tag LePetit2 jpg width gt 650 height gt 310 gt Blacklight takes care of the image s full path 4 2 4 Changing the content of the homepage Figure 9 depicts the current content on the Russian Flu homepage Welcome Welcome to the Russian Flu 1889 1890 The research will be done by a team of Virginia Tech undergraduates working in collaboration with a historian This site will provide updates on the project resources and findings Check out our Blog Russian Flu Blog Multimedia Timeline 4 Overall Flu Spread MAP Figure 9 Current Russian Flu homepage content 15 To change the homepage content edit the home text html erb in the blacklight blacklight app views catalog directory Figure 1 Figures 10 and 11 display the resulting Russian Flu website All Fields Search Limit your search como Newspaper Title Welcome to the Russian Flu Infection Location Reporting Location Language This project explores the relationship between the spread of disease and the spread of information during the Russian Flu epidemic 1889 1890 The research will be done by a team of Virginia Tech undergraduates working in collaboration with a historian This site will provide updates on the project resources and findings Check out our Blog Russian Flu
14. arching tools to best suit our client s requests We researched various ways to store and search data and decided Solr would be the most effective search mechanism We then had to decide on a front end to use that integrated with Solr We settled on using Blacklight an open source Ruby on Rails gem that provides a discovery for any Solr index In March we created a prototype of the user interface s design and function Another major decision regarded the configuration of the searching We decided to implement a faceted search option and began organizing the fields The final list of facet fields was Newspaper Title Infection Location Reporting Location Language Date and Keywords The majority of March consisted of working with the translation team associated with the project We had to explain the formats for each data field in order to index the articles into Solr and ensure that every required field had a value and the format of each category was valid and consistent A major alteration in the formats pertained to the storage of the date field Solr expected the dates in the format YYYY MM DDTHH MM SSZ YYYY MM DDT00 00 002Z therefore the translation team had to update their articles April 1st marked the setup completion of the Russian Flu server This allowed us to stop working on our local machines and move files onto the library server On April 28th the Russian Flu project team gave a presentation showcasing accomplishments thus far a
15. ashington WV Pea e Nashville TN Memphis Atlanta SC e Dallas GA LA e Houston e New Orleans e Lebanon Reading Harrisburg A Carlisle Norristown Lancaster f e Chambersburg Philadelphia camden e Wilmington e 4 Hagerstown Newark Martine il l e Frederick uw r e F c Columbia Annapolis Millville Dover Ee Arlington Washington Alexandria St Charles delphia Figure 30 StorvMapjS at specific data point More information on this resource e Vinelanc US MANIFEST DESTINY The mean center of U S population is determined by the United States Census Bureau from the results of each census Defined as the point at which an imaginary flat weightless and rigid map of the United States would balance perfectly if weights of identical value were placed on it so that each weight represented the location of one person on the date of the census This is an overview or title slide to show all the points in your story routed on your map Claimed by Spain Claimed by the U S Federal Censuses Exist Only For Those Areas In White ip of the US 1790 A look at the US during this first Census data collection Map left is an example of a URL to an image hosted on the web http storymap knightlab com 27 Lessons Learned Timeline Our client signed off on our project on February 37d and we spent the following two weeks rese
16. ate str str name facet field Tags Keywords str Figure 3 Default search result fields Figure 4 Facet fields in the solrconfig xml requestHandler lt for requests to get a single document use id 666 instead of q id 666 gt requestHandler name document class solr SearchHandler gt lst name defaults gt str name echoParams gt all lt str gt str names fl x str str name rows gt 1 lt str gt str name q gt raw f id v id lt str gt use id 666 instead of q id 666 lst requestHandler Figure 5 Document view search handler in solrconfig xml 12 4 Blacklight Blacklight is an open source Ruby on Rails user interface for Apache Solr Backlight uses Solr s search engine to query and display indexed data Blacklight includes the following features Faceted browsing Relevance based searching Bookmarkable items Permanent URLs for every item User tagging of items 4 1 Setting Up Blacklight To generate a Blacklight application please refer to Blacklight s Quickstart Guide https github com projectblacklight blacklight wiki Quickstart This tutorial will provide and index sample data See Figure 6 for the resulting default Blacklight application blacklight All Fields Limit your search Welcome Format To modify this text to your specifications copy this file located in the blacklight plugin dire
17. cated at the full path blacklight blacklight See Figure 1 for the file configuration within the blacklight directory 2 3 Initial Commands All work done within the Blacklight application should be done as the blacklight user To become the blacklight user issue the following command user sudo u blacklight s The Blacklight application is set to production mode so the server is always running In order to restart the server execute the following commands user sudo u blacklight service blacklight jettv Stop user touch tmp restart txt user sudo u blacklight service blacklight jetty start Note One must be in the tmp folder in order execute the touch command as is Otherwise redirect touch to the path to the restart txt file Working in production mode also means that any changes to a file will directly affect the website 3 Apache Solr Apache Solr is an open source full text search engine platform Here is a tutorial to set up Solr and configure its capabilities http lucene apache org solr tutorial html Solr indexes data with a consistent document definition defined in the schema xml file The search configuration is an HTTP interface with faceted search spelling suggestions and high relevancy results which is defined in the solrconfig xml file Solr resides within the Blacklight application s jetty directory See Figure 1 for Solr s file locations 3 1 Defini
18. cation gf Date Infection Location 50000 collector Reporting Location 25888 str str name location pf str name pf gt arm Infection Location 588888 qe de Reporting Location 258888 Translation 250088 pic a RR Infection Location 100008 qion guage_q Reporting Location 58888 Hsu 9 Language 25888 i Date 188 str name language pf collector 18 Language str str Figure 16 In solrconfig xml Specifies fields Figure 17 In solrconfig xml Specifies to include in default search fields fields to include in drop down menu 5 3 Results List The fields to display in the results view can be configured within the catalog controller rb file Figure 18 depicts the commands necessary for the fields in the results view in Figure 19 18 solr fields to be displayed in the index search results view The ordering of the field names is the order of the display config add index field Newspaper_Title label gt Newspaper Title config add index field Infection_Location label gt Infection Location config add index field Reporting Location label gt Reporting Location config add index field Date label gt Date date gt 4 format gt short l config add index field Language label gt Language config add index field Taqs Kevwords label gt Kevwords config add index field Translation label gt Translation Figure 18 De
19. ctory gluck mounts var Publication Year www html blacklight bootstrap3 releases 1391278869 vendor bundle ruby 2 0 0 bundler gems blacklight c6e582da6024 app views catalog home text html erb Topic to vour top level rails app gluck mounts var Language www html blacklight bootstrap3 releases 1391278869 app views catalog home text html erb Note that you will need to create the catalog directory under views Call Number Region Era Publish Date Figure 6 Blacklight s demo site 4 2 Customizing Blacklight All customization of the Blacklight instance should be done within the application i e within the blacklight blacklight directory Upon generating the Blacklight application an instance of Solr including the jetty server is included 13 Configuration of the Solr fields and Blacklight visualization including facets index fields search parameters and show fields will go in the catalog controller rb file See Figure 1 for the location of this file For more information on how to tailor the catalog controller please refer to the Configuring and Customizing Blacklight section in the Quickstart Guide https github com projectblacklight blacklight wiki Configuring and Customizing Blacklight For the purpose of the Russian Flu project our client approved the default layout Figure 7 Apart from the changes made to the search and facets we only needed to change the color scheme This can be done in th
20. d show field Reporting Location label gt Reporting Location config add show field Date label gt Date date gt true config add show field Language label gt Language config add show field Tags Keywords label gt Keywords config add show field Translation label gt Translation config add show field Image Link label gt Image Link Figure 21 Document fields defined in catalog controller rb 6 Site Maintenance The sections above discussed the initial creation and implementation of the current Russian Flu website Currently the Russian Flu website needs to be maintained and expanded per the client s request The following sections describe common scenarios and commands needed to maintain the website 6 1 Edit ONLY as the Blacklight User Developers should only make changes to the Blacklight application any file in the blacklight blacklight directory as the Blacklight User Upon accessing the Russian Flu server a developer s username is their PID To become the Blacklight user execute user sudo u blacklight s 6 2 Stop Start Server Upon changing files in the Blacklight application a developer must restart the server for the updates to appear To stop the server blacklight sudo u blacklight service blacklight jettv Stop To restart Phusion Passenger blacklight touch tmp restart txt To start the server blacklight sudo u
21. e css files provided within the public assets directory Figure 1 The logo image and favicon an icon associated with a URL that is variously displayed as in a browser s address bar or next to the site name in a bookmark list were changed for the website CSS is a style guide that formats the different html object ids given in the site upon creation Please refer to the CSS tutorial http www w3schools com css default asp 4 2 1 Changing the favicon and title of the webpage Figure 7 depicts the current favicon and title Ri Russian Flu x Figure 7 Current favicon and title To change the favicon replace favicon 82609ecd8ba4a1947d9a00bd11484cc8 ico in the blacklight blacklight public assets directory Note The new icon must be named the same and the size must be 16x16 32x32 48x48 or 64x64 pixels To change the title change Russian Flu in the line title 96 Russian Flu gt lt title gt in blacklight html erb in the blacklight blacklight app views layouts directory Figure 1 4 2 2 Changing the header image Figure 8 displays the current header image Che Russian Flu Pandemier 1889 1890 Figure 8 Russian Flu site logo To change the current header replace 14 logo f7591a2fac407a81fdb3457d8fdb5083 png in the blacklight blacklight public assets blacklight directory Note the new header image must be named the same 4 2 3 Adding images to the site Place the desired image in the bl
22. fines fields to show in results view in catalog controller rb u Previous 1 10 of 16 Next 1 Intelligenzblatt und Berner Stadtblatt intelligenzbiatt und Berner Stadtbiatt London Bern 1890 01 06T00 00 00Z German postmaster Salisbury La Lanterne Paris 1890 01 03T00 00 00Z 3 Intelligenzblatt und Berner Stadtbiatt Intelligenzblatt und Berner Stactbiatt Figure 19 Results view of the Russian Flu website 5 4 Document View A single document can be viewed by clicking on the Newspaper Title link Figure 19 The document view can be configured in the solr xml file Figure 20 and the catalog controller rb file Figure 21 lt requestHandler gt lt for requests to get a single document use id 666 instead of q id 666 gt requestHandler names document class solr SearchHandler gt lst name defaults gt str name echoParams gt all lt str gt str name z fl x str str name rows gt 1 lt str gt str name q gt raw f id v id str lt use id 666 instead of q id 666 lst lt requestHandler gt Figure 20 Document view search handler in solrconfig xml 19 solr fields to be displayed in the show single result view The ordering of the field names is the order of the display config add show field Newspaper Title label gt Newspaper Title config add show field Infection Location label gt Infection Location config ad
23. n tvpes long indexed true stored true required false gt name id tvpes text general indexed true stored true required true multiValued false name Newspaper Title tvpes text general indexed true stored true required false multiValued false gt name Infection Location tvpes text general indexed true stored true required false multiValued false name Infect Lat Long types text general indexed true stored true required false multiValued false gt name Reporting Location tvpes text general indexed true stored true required false multiValued false name Report Lat Long type text general indexed true stored true required false multiValued false gt name Date type date indexed true stored true required false multiValued false name Link tvpes text general indexed true stored true required false multiValued false name Translation tvpes text general indexed true stored true required false multiValued false gt name Language type text general indexed true stored true required false multiValued false name Tags Keywords tvpes text general indexed true stored true required false multiValued true gt name Image Link tvpes text general indexed true stored true required false multiValued false name Translator tvpes string indexed true stored true required false multiValued false name collector
24. nd future plans Our team spent the latter part of the semester troubleshooting various problems and fixing bugs some of which were not solved Problems and Solutions Throughout the course of our project we encountered various problems most of which we found solutions The first problem encountered pertained to the searching and storage of article metadata We researched many databases including SQL and mongoDB After meeting with Dr Fox we decided the best implementation included using Apache Solr Upon selecting Solr as a back end we needed to find a front end user interface in which to integrate Solr From our research we concluded that Blacklight would suffice in that there would not be a steep learning curve We also chose Blacklight because some of group members were familiar with Ruby on Rails Our next problem was getting access to server in which to host our website On April 1st we were granted access to a library server which allowed us to begin integrating Solr and Blacklight in a remote location Unfortunately gaining access to a server 28 this late in the semester hindered the amount of work we could accomplish and problems that were solved Another semester long issue was that of article data consistency The translation team was in charge of uploading data in the correct format The team used a Google Spreadsheet and often times cells were left blank not spell checked not format checked etc Given we indexed 2
25. ng Indexed Data Solr s schema is written in XML and defines the fields and field types for indexed data See Section 3 3 for the complete schema xml file 3 2 Common Tasks Please see the Section 7 Site Maintenance for the full steps on the following actions 3 2 1 Indexing Data Data is indexed through the computer terminal Currently data is uploaded in the format of a tab separated csv file through use of the curl command The process of uploading the translated article data is discussed below in Section 6 3 Site Maintenance Indexing Data 10 3 2 2 Clearing the data Upon altering Solr s schema xml or solrconfig xml files one needs to clear all of the data indexed in Solr After clearing the data the data needs to be re indexed The changes made to the schema xml and solrconfig xml files will then be apparent See Section 6 4 Site Maintenance Clearing Deleting Data for the full instructions 3 3 Schema xml Figure 2 depicts the full schema xml used to index the translated article data The first XML tag fields describes the list of fields to be stored in Solr The final XML tag types described the type of each indexed field i e The Newspaper Title field is a text general while the Date field is a date schema name example version 1 5 gt fields field field field field field field field field field field field field field field field names versio
26. rue add facet field Date label gt Date date gt format gt short add facet field Tags Keywords label Keyword Figure 12 Code in the catalog controller rb for facet display on website Limit your search name facet true str name facet mincount 1 str str name facet limit 18 str gt Newspaper Title name z facet field Newspaper Title str names facet field sInfection Locationt str names facet field gt Reporting_Location lt str gt names facet field gt Language lt str gt names facet field gt Date lt str gt name facet field Tags Keywords str the times Infection Location Figure 14 Code in solrconfig xml defining facet fields Reporting Location Language spanish Figure 13 Facet display on site 5 2 Drop down Menu 17 Using the field names defined in schema xml the following images depict the implementation of the drop down menu Figures 16 and 17 must be included in the solrconfig xml file in order for the drop down menu fields to appear y All Fields Newspaper Title Location Language Figure 15 List of defined drop down search fields str name gf str name newspaper qf Newspaper Title i1888008 B tnde Tags Kevwords 75800B l Translation 25088 str nempe newspaper pf Infection Location 10088 Newspaper Title 200 Reporting Location 5888 lt str gt l i Language 2580 str name lo
27. the desired information There are many different ways you can represent your timeline and many different types of information that can be included Steps to creating a timeline more detail in TimelineJS page 24 Create your spreadsheet from a template provided Publish spreadsheet to the web Copy spreadsheet URL into the box provided in the page Embed the code provided into the website pe DW P This YouTube video tutorial shows you how to create your timeline https www youtube com watch v vAWbm4gF9lU For the Russian Flu project we wanted a time lapse of how the flu had spread so we only decided to include images But YouTube videos can also be used and text included accompanying the images The Russian Flu spreadsheet for TimelineJS is displayed in Figure 26 Start Date End Date Headline A Graphical Representation of the Occurance of the Influenza Pandemic Across the Earth in 6 5 1 1889 0 00 00 10 31 1890 0 00 00 Different Time Spans From May 1889 to October 1890 httos scholar vt edt d2c7 467b 8609 8c34c8044809 1889 May through September 1890 20Map 20R 5 1 1889 0 00 00 9 30 1889 0 00 00 1889 September 201889 httos scholar vt edt d2c7 467b 8609 8c34c8044809 1889 10 1 1889 0 00 00 10 31 1889 0 00 00 October 1889 1890 20Map 20R httos scholar vt edt d2c7 467b 8609 8c34c8044809 1889 November 1st through 1890 20Map 20R 11 1 1889 0 00 00 11 15 1889 0 00 00 November 15th 1889 15 2C 201889 jpg ht
28. timeline knightlab com gt 31
29. tion team to format the data correctly in order to index it into Solr Our client Dr Ewing hopes to continue the Russian Flu project in the future as well as extend the website s functionality Articles will continue to be translated and article data will continue to be indexed Users Manual Searching There are two approaches to searching the database for translated articles The first is to use the search box and drop down menu To search within a particular field use the drop down menu and type the corresponding text in the search box The second is to use the facet fields One can also narrow down a search through using a combination of the search box drop down menu and facet fields Note The facets will update to the search parameters Limit your search Newspaper Title Infection Location Reporting Location Language spanish Links and Other Features The three links at the top of the webpage are Bookmarks History and Login You can bookmark an article by clicking on the bookmark button when you are on its page which currently you cannot get to All Fields Sal Search Bookmarks Cite Email 1 Fi vre dengue et grippe d apr s l pid mie de 1889 1890 Newspaper Title Fi vre dengue et grippe d apr s l pid mie de 1889 1890 Infection Location Paris Original Text http gallica bnf fr ark 12148 bpt6k5442899k f8 image r influenza9620la9620AND9620grippe9620 langEN 2
30. to meeting with a Ruby on Rails developer we have posted a Blacklight user forum addressing the linking issue Our group has gained much knowledge this semester the most obvious through working with novel applications Another lesson includes starting assignments earlier even if all resources are not available We also should have reached out to other groups working with similar applications to troubleshoot issues and ask questions Another important lesson learned was that of communication The ongoing formatting issues were hard to communicate to project members that did not have CS background knowledge This information gap proved to inhibit our accomplishments in that the formatting requests took more time than necessary to fix Using examples and ensuring group members had a full understanding of the desired formatting changes would have helped expedite the process of indexing article data 29 Acknowledgements We would like to express our appreciation to Tom Ewing for all of his helpful input and feedback during the implementation of the web database created for his Russian Flu project We would like to also thank him for providing all the information and content needed to make the website a more finished product We wish to thank various people for the contribution to this project The translation team for being part of the Russian Flu project and their hard work in providing us with the translations needed to fill the translations dat
31. tos scholar vt edu d2c7 467b 8609 8c34c8044809 1889 November 16th through 1890 20Map 20R 11 16 1889 0 00 00 11 30 1889 0 00 00 November 30th 1889 30 2C 201889 jpg httos scholar vt edt d2c7 467b 8609 8c34c8044809 1889 December 1st through 1890 20Map 20R 12 1 1889 0 00 00 12 15 1889 0 00 00 December 15th 1889 15 2C 201889 jpg httos scholar vt edt d2c7 467b 8609 Figure 26 Google Doc spreadsheet for TimelinejS Figures 27 and 28 depict the final Timeline See Figure 1 for the file location of the timeline 25 A Graphical Representation of the Occurance of the Influenza Pandemic Across the Earth in 6 Different Time Spans From May t Figure 27 Final Timeline front page November 16th through November 30th 1889 aprical Zepreserestion of me Occurance of the November 16th through November 30th 1889 Figure 28 Final Timeline structure For more information about TimelineJS and other views http timeline knightlab com 26 7 3 Other Multimedia Resources StoryMapjJS Knight Lab provides another tool StorvMapjS which allows one to add information to a map and go forward chronologically geographically Figures 29 and 30 depicta working version of StoryMapJS This will be a resource for expansion in the future Montr al Ottawa WI Milwaukee Detroit e PUE e Cleveland PA ew I e New as A XE ais dd Philade e Kansi CAPE Cincinnati Baltimore Louis W
Download Pdf Manuals
Related Search