Home

BIBLIOMETRIC ANALYSIS OF SCIENTIFIC AND TECHNOLOGICAL

1. tHe N DO DW G wo ON Hh yu nuenanmwoans le a OUTPUT lime ASAS The author file includes three fields and the key field This file shows the names of all authors for each document as well as each of the addresses A 8 APPENDIX II Specification of the database of the Observatoire des Sciences et des Technologies Figure C Sample entry of the author file Edit Dota Yiew Dustomizo Globale Options Window ek VIEWTABLE SANADA AUTCAN95 107000004 Mcdonad A 107000005 Yhap J 107000006 Milar BG 107000007 Poel Jv 107000008 Fiske JA 107000009 Debrujn E 107000009 Dudley D 107000010 JosephNB 107000011 Resula J 107000012 Issacs TL 107000013 Brernan S 107000014 Katerberg WH 107000015 Kukla A 107000016 Ruse M 107000017 Gillon BS 107000018 Trigger B6 107000019 Golombek L 107000019 Mason RB 107000020 Swain M 107000020 Lapkin S 107000021 Shandro A 107000022 Freedman A 107000023 Kilpatrick RS 361843 The address file contains a list of all the addresses linked to each document The information has been standardised for all Canadian institutions The address is used to encode the type of institution the city the administrative region the province as well as the country of origin of the authors who worked on the document APPENDIX II Specification of the database of the Observatoire des Sciences A 9 et des Technologies Figure D Sample entry of the address file VIEWTABLE C
2. LEYDESDORFF L The Science Citation Index and the measurement of national performance in terms of numbers of scientific publications Scientometrics Vol 17 Nos 1 2 1989 pp 111 120 LEYDESDORFF L Problems with the measurement of national research performance Science and Public Policy Vol 15 No 3 June 1988 pp 149 152 LEYDESDORFF L and E GAUTHIER The evaluation of national performance in selected priority areas using scientometric methods Research Policy Vol 25 1996 pp 431 450 MOED H F R E de BRUIN and TH N van LEEWEN New bibliometric tools for the assessment of national research performance Database description overview of indicators and first applications Scientometrics Vol 33 No 3 1995 pp 367 380 NOYONS E C M and A F J van RAAN Actor analysis in neural network research The position of Germany Research Evaluation Vol 6 No 2 August 1996 ROYAL SOCIETY Evaluation of national performance in basic research Study by the Royal Society Research Service for the Advisory Board for the Research Councils London 1986 46 Bibliography O HERLIHY J RT amp D regional development and evaluation Scientometrics Vol 34 No 3 1995 pp 513 518 SYLVAIN C Canadian research activity in aquaculture A bibliometric analysis Scientometrics Vol 27 No 3 1993 pp 295 316 TIJSSEN R J W T N van LEEUWEN and J C KOREVAAR Scientific publication activity of industry
3. an activity index for the major publishing countries as a group an international list of the most active institutions in this area according to the distribution of publications and patents a list of Canadian institutions that have published documents and their area of specialisation a list of Canadian institutions holding patents according to their area of specialisation Research managers were made aware of the state of research throughout the world and of specific research efforts in Canada Source R Dalp E Gauthier and M P Ippersiel The state of nanotechnology research Report presented to the National Research Council of Canada May 1997 Appreciating the significance of networks consortia and other forms of association in the innovation process for example helps gain a better understanding of how these associations can be advantageous for a business With better associations businesses are able to determine the most productive links in the network in terms of the latest research developments as well as the people who are best positioned within the network gt Thus research managers can put bibliometric data to good use in order to position a research unit strategically whether it is a university department a public laboratory or a business 25 W A Smith Evaluating research technology and development in Canadian industry Meeting the challenges of industrial innovation Scientometrics Vol 34 No 3 1995 53
4. June 1985 NEDERHOF A J The validity and reliability of evaluation of scholarly performance in A F J van Raan et al Eds The handbook of quantitative studies of science and technology Amsterdam Elsevier Science Publishers B V 1988 pp 193 228 NEDERHOF A J and A F J van Raan A bibliometric analysis of six economics research groups A comparison with peer review Research Policy 22 1993 pp 353 368 NEDERHOF A J R F MEIJER H F MOED and A F J van RAAN Research performance indicators for university departments A study of an agricultural university Scientometrics Vol 27 No 2 1993 pp 157 178 NEDERHOF A J and E C M NOYONS Assessment of the international standing of university departments research A comparison of bibliometric methods Scientometrics Vol 24 No 3 1992 pp 393 404 PEREIRA J C R M L PIRES P S DUARTE A T PAES and V OKANO Introducing a method of research evaluation into a university Medical research at the University of S o Paulo Brazil Research Evaluation Vol 6 No 1 April 1996 Van RAAN A F J Evaluation of research groups in D EVERED and S HARNETT Eds The evaluation of scientific research Chichester UK John Wiley and Sons 1989 RINALDINI C Experience on research evaluation at the Joint Research Centre of the European Commission Scientometrics Vol 34 No 3 1995 pp 519 525 SKEA J E F B R MARTIN and E N LING Assessing u
5. LAVILLLE Guide pratique pour l utilisation des donn es sur les brevets comme indicateurs scientom triques Paris OCDE 1992 SIMONETTI R et al Product and process innovations How are they defined How are they quantified Scientometrics Vol 32 No 1 1995 pp 77 89 SMITH K Technological innovation indicators Experience and prospects Science and Public Policy Vol 19 No 6 1992 pp 383 392 TEITEL S Patents R amp D expenditures country size and per capita income An international comparison Scientometrics Vol 29 No 1 1994 pp 137 159 7 3 Citation and co citation analysis This subsection contains a list of documents dealing with citation and co citation methods Small and Garfield are pioneers of citation analysis The papers by Small and Sweeney and by Small Sweeney and Greenlee provide a good description of co citation analysis A few empirical studies using one or the other of these methods have been added to illustrate the many possible applications The paper by Milman and Gavrilova deals with the study of one discipline Nederhof et al describe the use of citations for academic evaluation purposes Wallmark and Sedig establish a comparison between citation analysis and peer judgment as evaluation tools Much has been written about citation and co citation analysis because of the many possible applications and as a result of the many debates aroused by citation analysis This short bibliography wil
6. No 3 1991 pp 263 282 CLARKE Thomas E The evaluation of R amp D programs and personnel A literature review 1986 Stragate Consultants CUNION K M U K Government departments experience of RT amp D programme evaluation and methodology Scientometrics Vol 34 No 3 1995 pp 375 389 CUNNINGHAM P The evaluation of European programmes and the future of scientometrics Scientometrics Vol 38 No 1 1997 pp 71 86 DALP R and F ANDERSON L Evaluation des politiques scientifiques et technologiques Montr al Centre interuniversitaire de recherche sur la science et la technologie CIRST Notes de recherche 92 04 1992 GEORGHIOU L A DALE and H CAMERON Eds Special issue on national systems for evaluation of R amp D in the European Union Research Evaluation Vol 5 No 1 April 1995 pp 1 108 GONDA K and F KAKIZAKI Research technology and development evaluation Developments in Japan Scientometrics Vol 34 No 3 1995 pp 375 389 HILLS P PREST s experience of evaluation University of Manchester Programme of policy research in engineering science and technology Scientometrics Vol 34 No 3 1995 pp 401 414 IRVINE John and Ben R MARTIN Evaluating big science CERN s past performance and future prospects Scientometrics 7 1985 pp 281 308 KYRIAKOU D Macroeconomic aspects of S T programme evaluation Scientometrics Vol 34 No 3 1995 pp 451 459 44 Bibl
7. An Experimental Canadian Workplace and Employee Survey by Garnett Picot and Ted Wannell June 1996 Are the Costs and Benefits of Health Research Measurable by M B Wilk February 1997 Technology and Economic Growth A Survey by Peter Hanel and Jorge Niosi April 1998
8. Public Policy Methods and Examples Example 2 Specialization indices for Canada 1995 Clincial medicine Biomedical research Applied science and engineering Earth sciences Source Observatoire des sciences et des technologies CIRST March 1998 3 1 International flow patterns Exchanges between countries are undoubtedly the most best known form of knowledge flow To obtain a profile of a country s scientific collaboration with other countries for a given year the first step is to list all the publications in which the country in question is mentioned in the address field The next step is to compile all the co authorships with other countries displayed in the address field The countries are identified on the basis of the institutional affiliation of the authors The indicator itself may be relatively simple but listing the share of each country taking part in a publication remains a delicate operation There are in fact several ways of counting publications having more than one country in the address field The publication can be broken down into as many parts as there are countries involved For example each country taking part in a publication might be counted for 1 3 if the address field contains three different countries This is the so called breakdown method A second method consists in assigning the origin of a publication to the country of the first author This so called first author meth
9. Public Policy Methods and Examples Example 6 Average impact of Canadian publications In a feasibility study carried out for the Natural Sciences and Engineering Research Council Gingras used impact measurements to compare the average impact of Canadian publications in the fields of evolution and ecology to the average impact of international publications in the same fields The impact measurement was standardised previously An average impact value of 1 0 means that the Canadian output is equivalent to the international output A value of less than 1 0 means that the average impact of the Canadian output is lower or less visible than that of the world average 0 84 0 837 0 82 0 817 0 80 0 79 T Average Impact 0 78 T 0 77 7 0 76 0 75 1988 1993 Source Yves Gingras Bibliometric analysis of funded research A feasibility study Report to the Program Evaluation Committee of NSERC April 1996 In terms of evaluation the advantage of the impact factor is that unlike the number of times an article is actually cited it is made available each year through the Journal Citation Reports Citation analysis on the other hand requires an interval of about four years between publication of an article and the citation count If evaluation procedures are to be truly effective they must be carried out as soon as programs have ended As an evaluation tool the impact factor helps solve a number of problems There
10. Sciences A 5 et des Technologies The database of the Observatoire des Sciences et des Technologies can also be used to identify the sector of activity university hospital province industry other to which the authors of a publication belong These sectors were identified on the basis of the standardised addresses of the publications themselves Thus 99 3 of the publications were linked to a type of institution Table B lists the 16 categories of institutions Three sources were used to identify disciplines and areas of specialisation For articles from SCI the classification of disciplines established by Computer Horizon Incorporated CHI was used This classification includes 9 fields and 106 subfields As for the arts humanities and social sciences the classification used was that proposed by ISI A list of CHI and ISI classifications will be found in Appendix III Once the types of institutions have been identified it is possible to provide detailed analyses of their scientific activities and of their interrelationships This type of analysis can also be applied to specific industrial sectors since industries can be linked to a standard industrial classification codes SIC Table B Categories of institutions in the database of Canadian authors Sector Type of institution 1 University university 2 Hospital hospital veterinary hospital 3 Government federal government municipal or regional government provincial govern
11. Technologies However to avoid producing percentages of each country s contribution that exceed 100 the denominator used in the calculations is the sum of publications attributed to each country 3 2 National flow patterns The flow of knowledge occurs not only between countries but also within countries In fact co author analysis can help to determine links between provinces cities and institutional sectors Just as scientific collaboration can be studied internationally networks of co operation can be identified between provinces or even between actors within the same province This type of indicator is established by determining the province in each Canadian address By compiling the percentage of articles in a given province written in collaboration with authors from the same province or from another province it is possible to show in a given year the flow pattern within a province or between actors of different provinces i e inter provincial flow patterns Example 4 Regional collaboration the example of the Montreal area In a bibliometric study of regional scientific collaboration Godin and Ippersiel prepared a profile of the flow of knowledge between researchers in the Montreal are and the rest of Quebec They analysed 23 953 articles containing at least one address in the Province of Quebec The findings indicated that all regions have a tendency to collaborate more at the international level than with Montreal or Quebe
12. belief for research conducted by other types of actors For this reason they deserve a place in scientific and technological directories 1 1 Descriptive indicators Bibliometric indicators can be subdivided into two major categories descriptive indicators and relational indicators having an analytical function Listings of papers and citations listings of patents and the citations they contain are examples of the most current descriptive indicators They measure the volume and impact of research at various levels When they are used over prolonged periods of time they provide a means of identifying trends Enumeration methods are based on calculations of the number of scientific publications that can be attributed to one actor in a given area This may be an author an institution a sector of activity covering several institutions universities public laboratories industries or even a geographic area city province country A research area can be aggregated at the level of one scientific discipline or of one sub discipline one technology or even one specific technological niche Descriptive indicators can be applied to publications and patents depending on whether the analysis deals with scientific output or with technological output 1 2 Relational indicators Co author analysis is the most frequent relational indicator It helps identify links and interactions between the actors of national and international systems of s
13. evaluation process as well as for reflections on policies BAYEN M La veille technologique un enjeu national Revue frangaise de bibliom trie 1990 COURTIAL J P and J C REMY Toward the cognitive management of a research institute Research Policy Vol 17 1988 pp 225 233 COURTIAL J P M CALLON and A SIGOGNEAU The use of patent titles for identifying the topics of invention and forecasting trends Scientometrics Vol 26 No 2 1993 pp 231 242 DESVALS H and DOU H La veille technologique Paris Dunod 1992 DOU H L QUONIAM and P HASSANALY The scientific dynamics of a city A study of chemistry in Marseille from 1981 to the present Scientometrics Vol 22 No 1 1991 pp 83 93 HICKS D B MARTIN and J IRVINE Bibliometric techniques for monitoring performance in technologically oriented research The case of integrated optics R amp D Management Vol 16 No 3 1986 JAGODZINSKI S J P COURTIAL and B LATOUR How to measure the degree of independence of a research system Scientometrics 4 1982 pp 119 133 KOENING Michael E D A bibliometric analysis of pharmaceutical research Research Policy 12 1983 pp 16 36 MEADOWS A J and J G O CONNOR Bibliographic statistics as a guide to growth points in science Science Studies 1 1971 pp 95 99 50 Bibliography MILLER R and A MANSEAU Bibliometric indicators and the competitive environment of R amp D laborator
14. gathering materials and Francois Valli res INRS research professional who provided essential information concerning the database of the Observatoire des Sciences et des Technologies INTRODUCTION With the financial assistance of Statistics Canada the Observatoire des Sciences et des Technologies has set up a database used to generate bibliometric indicators This database of Canadian authors has been specifically designed to meet the needs of public policy makers and program administrators This guide to the methodology is above all a complement to Knowledge flows in Canada as measured by bibliometrics Tt shows how bibliometric studies can help understand the Canadian innovation system Particular emphasis has been placed on the flow of knowledge in Canada and more specifically on exchanges between countries provinces and institutional sectors For slightly more than 30 years now Western countries notably through the OECD have had access to indicators that help them monitor changes in national scientific and technological systems It was back in 1963 that OECD published for the first time the Frascati Manual proposing a sample method for surveys of experimental research and development The Manual provided a standard for government measures aimed at gathering information on investments in research and development The Frascati Manual has undergone several revisions and improvements since 1963 The Frascati Manual is based on the we
15. in mind the particular mission of the university before making comparisons between institutions The same constraints apply to the various administrative units within the institution as well as to various scientific disciplines In an effort to adapt to the realities of multi functional academic institutions bibliometric research uses convergent indicators to throw light on the multiple activities of administrative units The principal objective of convergent indicators is to overcome the limits of bibliometric indicators taken individually so as to create a set of partial indicators that are representative of all activities When several different indicators converge towards the same conclusion that conclusion is more valid 5 Bibliometric monitoring of research Bibliometric monitoring makes it possible to establish a quantitative profile of the state of research and therefore of national performance Public policy makers need to be constantly informed about overall research activities at the national level and about a country s relative position on the world scene for a given research activity Descriptive indicators provide such a monitoring at various levels of data aggregation Bibliometric monitoring is usually carried out at a fairly high level of aggregation such as national scientific output by discipline or by institutional sector However bibliometric monitoring can also be used for administrative monitoring in which case it deals mos
16. in the Netherlands Research Evaluation Vol 6 No 2 August 1996 4 4 Evaluating public institutions The evaluation of public institutions covers several issues The institutions in question may be public laboratories universities or research groups In each case the challenge is different The terms of reference of public laboratories may be based on pure research or on applied research for example Likewise evaluation procedures in a university setting may concern the institution as a whole one faculty one department or even one research group Bibliometric evaluations must keep in mind the particular characteristics of the institution The contributions included here cover a wide selection of institutions and national experiences public laboratories in Canada of course but also in Germany as well as universities in the United Kingdom Australia Greece Chile and Holland Some very specific cases are dealt with the Max Planck Institute in Germany technical universities in the United Kingdom and also the Joint Research Centre of the European Commission Readers who wish to know more about the evaluation of university research will read with interest the contributions by Martin and Skea and by Nederhof and Noyons The former provides a critical review of the potential of bibliometrics the latter compares various methods Note the originality of the paper by Nederhof and Noyons who propose a means of comparing university departments a
17. it might be useful to briefly describe the principal functions of bibliometric analysis applied to scientific and technological research One of the primary goals of scientometrics and bibliometrics applied to public policy is to serve as an information tool for decision making It is within such a context that the present document was written The objective is to provide public policy makers with a guide to the use of bibliometric tools currently available There are many applications of bibliometrics including the development and evaluation of science and technology programs and policies the management of private and public research or technological monitoring and strategic decision making The following then is a methodological guide to bibliometrics applied to public policy This guide deals with bibliometric indicators and methods within specific applications The author begins with a brief outline of descriptive and relational indicators This typology is based on a 1997 OECD document The principal indicators are then described within the broader context of their utilisation The emphasis is on the three main functions of scientometrics and bibliometrics i e description evaluation and monitoring of science and technology Examples of applications are provided for each topic The author provides an annotated bibliography of the various applications of bibliometrics The bibliography includes reference works on scientometrics and bibliometr
18. publishing activities at the level of countries provinces cities or institutions and is used for comparative analyses of productivity The data can then be used to assess the performance of research units as a complement to standard evaluation procedures Bibliometric data are also used as a benchmark for the monitoring of science and technology since longitudinal studies of scientific output help identify areas of research that are developing or regressing Table 1 Grid used for the conceptual analysis of bibliometric studies year EEN i ie NA ail Lf a Table 1 provides a schematic outline of the three levels of analysis of variables which can be applied to the database of the Observatoire des Sciences et des Technologies Thus the production of a single researcher can be measured within the framework of an evaluation This level is mostly used in conjunction with other evaluation methods since bibliometric tools are not considered a valid method of measuring the productivity of individual researchers The other two levels meso and macro are by far the most often used At the meso level bibliometric indicators describe the scientific production of institutions and research groups They can also be linked to grant programs for program evaluation purposes At the macro level bibliometric indicators are used to measure national output by country province or city National output is analysed as a means of comparing research
19. structured as well as the relationships between the different files Figure A How a relational database operates ARTICLE FILE AUTHOR FILE author 1 author 2 Document number Title of article Name of journal Discipline Impact factor Volume Final page ADDRESS FILE Language Type of document Total number addresses Total number authors name of institution institution Starting page SIC code city regional code province country Total number citations The article file comprises 12 searchable fields This file contains complete information on the bibliographic reference of each document The entry number for the database is the document number This is also the key that identifies the document in all three files The article file also provides the title of the document In the case of a journal article for example the entry contains the title of the article the title of the journal in which the article was published the volume and issue of the journal as well as the APPENDIX II Specification of the database of the Observatoire des Sciences A 7 et des Technologies starting page and the final page of the article The article file also shows the total number of authors the total number of addresses as well as the total number of references linked to the document This type of information is included as a guide Detailed information is provided in the address file and the author file Figure B Sam
20. systems and determining the links between the various institutions of a given national system 1 Bibliometric indicators In order to make rational decisions public policy makers need to have a firm understanding of scientific and technological activities Bibliometric indicators provide the only overall picture of the scientific 9 Xavier Polanco Infometrie et ing nierie de la connaissance in J M NOYER Ed Les sciences de l information bibliometrie scientom trie infom trie Rennes Presses universitaires de Rennes 1995 10 Bibliometrics Applied to Public Policy Methods and Examples output ofa country In a research paper dealing with science and technology indicators Godin described the present status of bibliometrics There may have been a time when the fact that bibliometric indicators were standardised limited their usefulness but this is no longer the case Furthermore they are not expensive to produce They do have their limits notably because they normally include only the natural sciences engineering and the biomedical sciences There is also an obvious linguistic bias that largely limits the coverage of scientific output to publications in English Finally it must be remembered that publishing represents only one of the activities of researchers In spite of such limits bibliometric indicators are one of the principal tools for measuring research output while providing a very good tool contrary to popular
21. the use of citations as an indicator of quality Nevertheless the citation indicator was never totally invalidated In an article on bibliometric techniques used to evaluate research programs Francis Narin President of Computer Horizons Inc explained the basic principles of citation analysis applied to research evaluations 1 publications especially scientific articles are a legitimate indicator of research productivity and 16 John de la Mothe The political nature of science and indicators Science and Public Policy Vol 19 No 6 December 1992 403 17 M Callon J P Courtial H Penan La scientom trie Paris Presses universitaires de France Que sais je 2727 1993 103 18 Ibid 110 Bibliometrics Applied to Public Policy Methods and Examples 19 2 references to articles the number of times reference is made to an article in another article published subsequently are a legitimate indicator of the quality or impact of the article cited Narin maintained that a comparison between traditional evaluation methods and bibliometric methods showed convergent results There would seem to be a correlation between article citation rates and the formal evaluation of articles by peers between researcher publication or citation rates and independent measurements of prestige such as scientific awards and between the publications of departments and the ranking of these departments by peers 2 Nowadays there is gre
22. they are supplemented by other databases The ISI databases do have certain limits but they will never completely invalidate the results of a bibliometric study Experienced analysts always take into consideration the inherent limitations of data 33 Cited in Leclerc Michel Y Okubo L Frigoletto and J F Miquel Scientific co operation between Canada and the European Community Science and public policy Vol 19 No 1 February 1992 p 16 34 Compendium 1996 Indicateurs de l activite scientifique et technologique au Qu bec Bureau de la Statistique du Qu bec 1991 p 43 See also Gingras Y C M daille La langue de publication des chercheurs qu b cois en sciences naturelles g nie et sciences biom dicales 1980 1988 Qu bec Minist re de l Enseignement sup rieur et de la Science Direction du d veloppement scientifique 1991 35 Cf E Garfield How ISI selects journals for coverage Quantitative and qualitative considerations Current Contents 22 1990 5 13 36 Patents and communications can be found in specialised databases Index to Scientific amp Technical Proceedings ISTP surveys conference proceedings in the fields of science and technology and has done so since January 1991 US Patent Search Claims amp Abstracts has been covering American patents since 1979 Bibliometrics Applied to Public Policy Methods and Examples 29 whenever they are interpreting results The database of the Observatoire des Sci
23. universitaires de France Que sais je n 2727 1993 COURTIAL J P Introduction la scientom trie De la bibliom trie la veille technologique Paris Anthropos Economica 1991 LAWANI S M Bibliometrics Theoretical foundation methods and applications Libri 31 1981 pp 294 315 NOYER J M Ed Les sciences de l information Bibliom trie scientom trie infom trie Rennes Presses universitaires de Rennes 1995 Van RAAN A F J et al Eds The handbook of quantitative studies of science and technology Amsterdam Elsevier Science Publishers B V 1988 Van RAAN A F J et al Eds Science and technology in a policy context Select Proceedings of the Joint EC Leiden Conference on Science and Technology Indicators Leiden The 34 Bibliography Netherlands 23 25 October 1991 Leiden DSWO Press Science Studies Series No 8 1992 WEINGART P et al Eds Representations of science and technology Proceedings of the International Conference on Science and Technology Indicators Bielefeld Germany 10 12 June 1990 Leiden DSWO Press Science Studies Series No 6 1992 2 Bibliometric statistics This section contains directory type documents Each one provides statistics on one or several aspects of scientific research The articles by Braun and colleagues are of particular interest They are part of a long term project aimed at documenting various aspects of scientific research throug
24. 0 Bibliography 35 BRAUN T and A SCHUBERT Indicators of research output in the sciences from 5 central European countries 1990 1994 Scientometrics Vol 36 No 2 1996 pp 145 166 EUROPEAN COMMISSION The European report on science and technology indicators Brussels 1994 EUROPEAN COMMISSION Research and Development Statistics 1995 Luxembourg Eurostat 1995 GL NZEL W A bibliometric approach to social sciences National research performances in 6 selected social science areas 1990 1992 Scientometrics Vol 35 No 3 1996 pp 291 307 LECLERC Michel Mesurer la science Dynamiques et mesures de la coop ration scientifique internationale du Quebec et du monde 1980 1990 Quebec Direction des politiques scientifiques et technologiques Direction g n rale des politiques industrielles scientifiques et technologiques minist re de l Industrie du Commerce de la Science et de la Technologie September 1995 MIQUEL J F T OJASOO Y OKUBO A PAUL and J C DORE World science in 18 disciplinary areas Comparative evaluation of the publication patterns of 48 countries over the period 1981 1992 Scientometrics Vol 33 No 2 1995 pp 149 167 MINISTERE DE L ENSEIGNEMENT SUPERIEUR ET DE LA SCIENCE MESS Compendium 1996 Indicateurs de l activit scientifique et technologique au Qu bec Qu bec Gouvernement du Qu bec 1996 NATIONAL SCIENCE FOUNDATION Science and engineering indicators 1996 Wash
25. 1 2 5 3 7 4 9 6 Italy 0 8 1 4 1 7 3 1 1 8 3 3 3 3 4 Australia 3 7 1 4 3 6 2 4 2 9 2 3 1 7 Switzerland 1 4 1 4 1 9 2 3 0 9 2 5 2 1 4 1 Sweden 1 7 1 8 1 7 2 5 1 4 0 1 1 7 Source Leclerc M La coop ration scientifique internationale lexemple du Canada La recherche Vol 24 No 252 March 1993 350 357 Each method has its advantages and disadvantages The method which assigns a publication to each of the participating countries is the method most often used to measure national performance It is the 13 Much has been written on the subject of counting publications and the study of national performance Cf Schubert A W Glanzel and T Braun Scientometrics data files A comprehensive set of indicators on 2 649 journals and 96 countries in all major fields and subfields 1981 1985 Scientometrics Vol 16 No 1 6 1989 p 7 J Anderson er al On Line Approaches to Measuring National Scientific Output A Cautionary Tale Science and public policy Vol 15 No 3 1988 53 161 L Leydesdorff Problems with the measurement of national research performance Science and public policy Vol 15 No 3 June 1988 149 152 D Lindsey Production and citation measures in the sociology of science The problem of multiple authorship Social studies of science Vol 10 1980 pp 145 162 16 Bibliometrics Applied to Public Policy Methods and Examples method that was used for the database of the Observatoire des Sciences et des
26. 1991 4 2 Program evaluation When scientific and technological activities came to be seen as public assets having an important role to play in the national economy there was a growing need to evaluate research programs because of their strategic significance Nowadays the evaluation criteria are legitimacy and relevance Traditional program evaluation methods such as peer assessment or client assessment for public laboratories for example are not always well suited to the new objectives The new evaluation criteria must be more objective and rigorous Moreover public policy makers need a general overview which a cost benefit analysis for example cannot provide Institutionalisation of the evaluation process has thus created a 22 Bibliometrics Applied to Public Policy Methods and Examples need for new evaluation tools and for more suitable databases This has opened the way to bibliometric methods Example 8 Convergent indicators in universities Ben Martin and John Irvine have created a model for the evaluation of scientific research that is based on input and output The method is used to establish a relationship between input and output so that research units can be compared to one other This evaluation model based on four elements uses the method of convergent indicators Anderson has provided the following summary 1 Identify and evaluate the input researchers financial and technical resources and the output contr
27. 3 Bibliometrics Applied to Public Policy Methods and Examples 25 Let us take for example a business operating in a high technology area such as biomedical research A bibliometric monitoring of its field of specialisation might help such a business understand different aspects of scientific output e g research topics that have been developed those that are emerging as well as the people doing the research Moreover analysing scientific publications and patents simultaneously would help the business understand specific scientific and technological developments within its field of specialisation Information gained through technological monitoring would therefore represent a comparative advantage for the business Co citation analysis and co word analysis are also tools that help research administrators understand scientific output in a broader context Co citation analysis determines the links between articles on the basis of citations A link is assumed to exist between two articles when they are simultaneously cited by a third article The strength of the link between the co cited articles is proportional to the number of articles that cite them simultaneously This makes it possible to identify networks of articles and authors Such networks provide clues to the cognitive development of specialities and disciplines They help pinpoint new research topics and track links between disciplines and the changes they undergo Co word analysis i
28. 87 102 MOED H F W J M BURGER J G FRANKFORT and A F J van RAAN The application of bibliometric indicators Important field and time dependent factors to be considered Scientometrics Vol 8 Nos 3 4 1985 pp 177 203 NARIN F and K HAMILTON Bibliometric performance measures Scientometrics Vol 36 No 3 July 1996 pp 293 310 NATIONAL BOARD OF EMPLOYMENT EDUCATION AND TRAINING Research Performance Indicators Commissioned Report No 21 Canberra Government Publishing Service 1993 OECD Quantifying technological advance S amp T indicators at the OECD challenges for the 1990 s Science and Public Policy Vol 19 No 5 October 1992 pp 281 290 OECD Manual on the measurement of human resources devoted to science and technology Canberra Manual Paris OECD 1995 OECD Proposed standard practice for surveys of research and experimental development Frascati Manual Paris 1994 OCDE R amp D statistics and output measurement in the higher education sector 1989 OECD Technology Economy Productivity Programme TEP Paris OCDE 1992 OECD and Eurostat Proposed guidelines for collecting and interpreting technological innovation data Oslo manual Paris 1997 Van RAAN A F J N J NEDERHOF and N F MOED Eds Science and technology indicators Their use in science policy and their role in science studies Select Proceedings of the First International Workshop on Science and Technology Indica
29. ANADA ADRCANSS il 29232 108016210 ECOLE POLYTECH DEPT en HER amp GENIE MONTREAL H3C 347 u PES 109015211 ECOLE POLYTECH DEPT GENIE ELECT amp GENIE MONTREAL i A HBC 347 U 20234 109016212 ECOLE POLYTECH DEPT GENIE ELECT amp MONTREAL Pa CANADA H30 347 U _29235 108014213 ETRANGER FRANCE 29236 108015213 ETRANGER FRANCE 29237 109015213 MCGILL UNM DEPT ELECT ENGN MONTREAL PQ CANADA H3A247 U 2238 103915214 ECOLE POLYTECH DEPT GENIE ELECT amp GENIE MONTREAL PO CANADA H30 347 U 29239 103916215 ETRANGER ITALY 29240 108015215 VICTORIA UNIV DEPT ELECT COWP ENGN VICTORIA BC CANADA VEW 3F6 U 29241 109015216 ECOLE POLYTECH DEPT GENIE ELECT LGENIE MONTREAL pa CANADA H3C347 U 29242 108015217 ETRANGER POLAND 29243 108015217 WICTORIA LAF DEPT ELECT amp COWPENGN ACTORIA BC CANADA VEW JP6 U il 292 103016218 NORTEL TECHNOL MONTREAL pa CANADA HISIS E 29245 108015219 NFOMAGNET TECHNOL CORP VANNIPEG VE CANADA E 29 100015219 UNV MANITOBA DEPT ELECT amp COWPENGN WINNIPEG va CANADA RST 2N2 U 29241 103015220 QUEENS UNM DEPT ELECT ENGN KINGSTON ON CANADA K7L3N5 U _ 29248 103016220 ROVAL ML COLL CANADA DEPT ELECT amp COWPENGN KINGSTON ON CANADA IKSU 29249 103015229 UNV MANITOBA DEPT ELECT amp COMP ENGN VANNIPEG MB CANADA R3T 2N2 u 2250 105015230 NATLRES COUNCI CANADA WANNIPEG Ma CANADA A3B 196 F 29251 108015230 UNY MANITOBA WANNIPEG vB CANADA R3T 2N2 U 1252 109015231 UMY TORONTO DEPT MECH ENGN TORONTO ON CANADA u 2925 103116252 ETRANG
30. BIBLIOMETRIC ANALYSIS OF SCIENTIFIC AND TECHNOLOGICAL RESEARCH A USER S GUIDE TO THE METHODOLOGY by Elaine Gauthier Observatoire des Sciences et des Technologies CIRST 88F0006XPB No 8 Science and Technology Redesign Project Statistics Canada September 1998 ST 98 08 THE INFORMATION SYSTEM FOR SCIENCE AND TECHNOLOGY PROJECT The purpose of this project is to develop useful indicators of activity and a framework to tie them together into a coherent picture of science and technology in Canada To achieve the purpose statistical measurements are being developed in five key areas innovation systems innovation government S amp T activities industry and human resources including employment and higher education The work is being done at Statistics Canada in collaboration with Industry Canada and with a network of contractors Prior to the start of this work the ongoing measurements of S amp T activities were limited to the investment of money and human resources in research and development R amp D For governments there were also measures of related scientific activity RSA such as surveys and routine testing These measures presented a limited and potentially misleading picture of science and technology in Canada More measures were needed to improve the picture Innovation makes firms competitive and more work has to be done to understand the characteristics of innovative and non innovative firms especially in the service s
31. ER DEPT ELECT ENGN USA 29254 108015232 UNV OTTAWA OTTAWA ON CANADA u 29255 103915235 BATISHCOLUMEIL CANC RES VANCOUVER BC CANADA VSZIL3 P 29256 108015234 ECOLE POLYTECH DEPT ELECT amp COWPENGN MONTREAL PQ CANADA H3C 347 U MESITA 10901575 UM Y MONTREAL DEPT PHYS MONTREAL PO CANADA H3CAL U 29258 109016236 ETRANGER USA 29259 109015236 ETRANGER ITALY 29260 108015236 TRIUMF VANCOUVER BC CANADA YBT 2A3 U of 4 F AA I Bass APPENDIX III Classification of scientific disciplines Discipline Biology Agricult amp food sci Botany dairy amp animal sci Ecology Entomology General biology General zoology Marine bio amp hydrobi Miscellaneous biol Miscellaneous zool Discipline Biomedical research Anatomy amp morphology Biochem amp molec biol Biomedical enginrng Biophysics cell biol cyt amp hist Embryology Genetics amp heredity genrl biomedical res Microbiology Microscopy misc biomedical res Nutrition amp dietet Parasitology Physiology Virology Discipline Chemistry Analytical chemistry Applied chemistry General chemistry Inorganic amp nucl chm Organic chemistry Physical chemistry Polymers Discipline Clinical medicine Personality amp soc ps Addictive diseases Allergy Anesthesiology Arthritis amp rheumat Behavioral science Cancer Cardiovascular systm Clinical psychology Dentistry Dermat amp venerl dis Devel amp child psycho Endocrinology Ex
32. LARD E Development of a method for detection and trend analysis of research fronts built by lexical or co citation analysis Scientometrics Vol 30 No 1 1994 pp 333 351 Citation Field of research Reference APPENDIX I Glossary When a document A mentions a document B document B is said to be cited as a source of information by document A In this sense all citations are derived from references Whereas the term reference is applied to the source of information the term citation emphasises the fact that a reference is cited or quoted The concept of citation is therefore purely analytical A field of research is the set of research subjects brought together under one topic Thus a field of research can be interpreted as an aggregate of scientific journals that deal with related research topics or that cite or quote each other In the latter case journal to journal citations are interpreted as an indicator of some affinity between the research subjects A field of research is therefore a series of research projects dealing with interrelated subjects A field of research is more or less consistent depending on the density of links between the research subjects References are displayed in the footnotes and in the bibliography of a given document The term reference is therefore applied exclusively to documents that another document uses as a source of information giving rise to a bibliographic description The reference
33. al specialised indices could have been used as a bibliographic source including Chemical Abstracts INSPEC PASCAL to name only the best known The choice of the ISI databases is justified by the fact that they are the only ones that contain the list of all addresses indicated in the publications This is the unique characteristic that led to the choice of ISI databases to set up the database The study of 26 H Small And F Greenlee Clustering the Science Citation Index using co citations I A comparison of methods Scientometrics Vol 7 1985 391 409 H Small And F Greenlee Clustering the Science Citation Index using co citations II A comparaison of methods Scientometrics Vol 8 1985 331 340 27 M Callon J P Courtial W A Turner and S Bauin From translations to problematic networks An introduction to co word analysis Social Science Information Vol 2 1983 191 235 M Callon J Law and A Rip Mapping the dynamics of science and technology London MacMillan 1986 26 Bibliometrics Applied to Public Policy Methods and Examples knowledge flow patterns between the participants of a given system requires complete data on the institutional affiliation of the authors However no direct link can be made between addresses and authors on an individual basis In fact the information is not presented methodically The first address does not necessarily match the address of the first author Likewise the number of addresses can b
34. aphic databases available on the market The initial step was used to retrieve all documents having the word Canada in the address of publications covered by SCI SSCI and AHCI for the years 1995 and 1996 These databases produced by the Institute for Scientific Information ISI are available in hard copy on compact disc CD ROM and on line They are all annual databases that are updated regularly i e monthly for the on line version and quarterly for the compact discs The CD ROM version was used to establish the database of Canadian authors Each of these databases covers several fields of scientific output Thus SCI covers all the areas of natural science engineering biology and medicine SCI available on CD ROM since 1973 includes each year between 3 500 and 4 000 scientific journals considered among the most representative of the literature 4 The SSCI database is also available as of 1973 it regularly covers over 1 400 journals To this must be added 3 100 journals in the natural sciences which are reviewed on a selective basis The AHCI database started in 1978 covers publications in the arts and humanities and indexes approximately 7 100 important journals including about 1 400 which are fully indexed and 5 700 from SCI and SSCI which are indexed selectively All types of documents covered by ISI are included in the database Overall the database of Canadian authors contains 24 types of documents from the three source
35. ater focus in bibliometrics on measuring the impact of research on the scientific community The concept of impact is related mostly to the dissemination of knowledge and less to the quality of research The number of references to an article is interpreted as a sign of influence or visibility in the scientific community There is wide consensus about this new interpretation The impact factor is an index used to measure the probable impact of research findings published in a scientific journal This index is related above all to journals The Journal Citation Reports published each year by ISI define the impact factor of a journal as the ratio between the citations and the published articles The impact factor of a journal for a given year is the total number of citations obtained for that year by articles published in the journal during the two previous years divided by the total number of articles published by the journal during the same two years The higher the prestige of a journal the higher the impact factor and the greater the probability that an article published in the journal will be cited As a result the impact factor is an indicator of the overall visibility of a journal and of its impact on the scientific community As applied to articles the impact factor indicates the number of times an article in a scientific journal will probably be cited on the average during a given period of time 19 F Narin Bibliometric techni
36. ational innovation system By breaking down the information contained in the addresses it is easy to gather the data needed for flow analysis The database provides information about the flow of international knowledge the flow of knowledge within and between provinces as well as the flow of knowledge between institutional sectors Co authorship is the preferred indicator used to describe collaboration and co operation in all areas of research Such collaborative efforts or flow lead to publications within the formal network of scientific journals These publications are the raw materials for the database Co author analysis is based on the principle that when two or more researchers jointly sign a paper intellectual and or social links can be assumed to exist between them Such links are measured by co author analysis An institutional or geographic analysis using co authorships shows the exchanges that occur between various countries or various segments of a national innovation system Flow analysis makes use of any and all information present in the addresses of authors Linking such information to the discipline associated with the corresponding journal provides a detailed picture of the many aspects of scientific exchanges Co author analysis thus helps identify the principal partners of research activities while providing a detailed picture of the formal network of collaboration within which exchanges take place 14 Bibliometrics Applied to
37. ations e the correlation between research expenditures and the subsequent publication volume the increase in the volume of publications supported by NIH with respect to the total increase of publications in the bibliometric database increases in NIH financial support for basic research as compared to that for clinical research publications supported by NIH in comparison with total publications broken down by specific field the percentage of publications supported by NIH appearing within the upper decile of articles cited in each field e the interaction among researchers according to the field and the source of funding The output indicators used in the evaluation procedure are the following publications standardised on the basis of journal impact publications standardised on the basis of author status citations output input ratio number of citations divided by the number of publications performance of the upper decile 10 of the most cited articles Percentage of articles published in a given field by researchers in a given institution as compared to the percentage of articles appearing in the upper decile in terms of citations Source Henry Small An analysis of research publications supported by NIH 1973 80 cited in H Small and E Smith Aversa An Agreement Between The Medical Research Council of Canada and The Institute for Scientific Information Philadelphia Institute for Scientific Information July
38. ator Scientometrics Vol 35 No 1 1996 pp 3 13 MELIN G and PERSSON O Studying research collaboration using co authorships Scientometrics Vol 36 No 3 July 1996 pp 363 378 40 Bibliography MIQUEL J F and Y OKUBO Structure of international collaboration in science Part 2 Comparisons of profiles in countries using a link indicator Scientometrics 29 No 2 1994 pp 271 297 MOED H F R E de BRUIN and A STRAATHOF Measurement of national scientific cooperation in CEC related areas of science during 1985 90 Report EUR14581 Office for Official Publications of the European Communities Luxemburg 1992 OKUBO Y J F MIQUEL I FROGOLETTO and J C DORE Structure of international collaboration in science Typology of countries through multivariate techniques using a link indicator Scientometrics 25 No 2 1992 pp 321 351 PETERS H P F and A F J VAN RAAN Structuring scientific activities by co author analysis An exercise on a university faculty level Scientometrics Vol 20 No 1 1991 pp 235 255 QIU Liwen A study of interdisciplinary research collaboration Research Evaluation Vol 2 No 3 December 1992 pp 169 175 SUBRAMANYAM K Bibliometric studies of research collaboration A review Journal of Information Science Vol 6 1983 pp 33 38 Bibliography 41 4 Evaluation 4 1 Research evaluation The documents listed in this section describe several studies on resear
39. because of the methodological limits of bibliometrics In fact bibliometric statistics are reliable only for sufficiently aggregated data That is why bibliometric tools are more robust when they are applied 24 According to Garfield citation analysis as an indicator of individual performance is inadequate The evaluation must take into account publication practices and citation conventions including self citation as they occur in the researcher s environment as well as his or her age and the prestige of the journals Eugene Garfield How to use citation analysis in faculty evaluations and when is it relevant Essay of an information scientist Vol 6 354 372 Nowadays citations are no longer used as an indicator of quality but rather as an indicator of research impact Bibliometrics Applied to Public Policy Methods and Examples 23 to administrative units meso level instead of individuals micro level The quality of an evaluation therefore depends on the unit being evaluated When an evaluation is carried out on the output of a university as a whole the objectives and multiple tasks of the institution must be kept in mind Research cannot be evaluated in the same way as teaching or the humanities in the same way as hard science As a university or department becomes more oriented towards teaching and applied research there is less likelihood that the volume of its publications will be significant It is therefore important to keep
40. c City The collaboration between regions is as follows Articles Co authorships Montreal between regions co authorships A B C C A Montreal 16 159 924 5 6 Montreal outskirts 1 318 436 32 2 292 67 0 22 2 Corridor 1 934 451 22 7 266 59 0 13 4 Quebec City 3 959 631 15 5 340 53 9 8 46 Regional outskirts 583 215 34 7 96 44 7 16 47 Source B Godin and M P Ippersiel Scientific collaboration at the regional level The case of a small country Scientometrics Vol 36 No 1 1996 63 The same method can be used to evaluate the flow pattern between institutional sectors Co author analysis applied to sectors is based on identifying the sector for each of the institutions contained in the Bibliometrics Applied to Public Policy Methods and Examples 17 address field The next step consists in counting for a given sector all the co authorships linked to each of the other sectors 5 Example 5 Collaboration within and between sectors in Quebec publications 1980 1990 Sectors University Hospital Industry Government University 8407 1771 179 612 Hospital 2374 37 72 Industry 444 41 Government 982 Godin analysed the publications of Quebec authors between 1980 and 1990 in order to verify the following hypothesis the proportion of publications from universities is decreasing whereas the share of other sectors is on the increase A sector of activity was assigned to each pub
41. c Information in Philadelphia ISI While the contribution of the United States in this respect representing 33 4 of the world s scientific output in 1990 might appear as an anomaly the United Kingdom continues to produce 7 4 of scientific information in spite of its declining influence during the last decade outperforming the Soviet Union Japan the Federal Republic of Germany and France Canada comes next which in view of its low investment equal to 10 billion according to 1992 forecasts is not so bad What is the explanation for this contradiction between human and financial investments and the apparently good results While an analysis of the authorship of Canadian articles may not provide an explanation it does give us a clue With almost 6 of internationally co authored articles Canada is ranked fifth among countries in terms of international co operation ahead of Japan but behind Sweden Italy France and Germany Canada invests little produces a lot in relative terms and co operates enormously Canada s international scientific co operation with its 10 first partners by discipline and percentage 1990 Biology Biomedicine Engineering Clinical Earth Math Chemistry Physics and technology medicine and space U S A 45 8 50 8 38 6 49 6 39 2 39 9 29 7 32 5 U K 7 8 8 1 5 5 6 7 13 5 8 9 6 7 France 3 1 9 1 5 7 7 1 5 8 9 4 9 7 Germany 3 8 5 2 7 2 2 5 3 4 9 7 7 9 Netherlands 3 7 3 2 1 4 5 4 5 2 5 4 9 4 5 Japan 2 3 2 7 5 5 3
42. c past performance analysis and peer judgement Scientometrics Vol 8 No 3 4 1985 pp 149 159 MOED H F W J M BURGER J G FRANKFORT and A F G van RAAN On the measurement of research performance The use of bibliometric indicators Leiden Leiden University Research Policy Unit 1983 NARIN F Subjective vs bibliometric assessment of biomedical research publications NIH program evaluation report US Department of Health and Human Services Public Health Service National Institutes of Health April 1983 VINKLER P Some aspects of the evaluation of scientific performances and related performances of individuals Scientometrics Vol 32 No 2 1995 pp 109 116 4 2 Program evaluation This section continues to deal with research evaluation though in the context of program evaluation Again particular attention is brought to bear on the experience of various countries The documents selected include papers on the European Community Japan the United Kingdom and Germany A number of contributions e g by Leydesdorff Moed Mullins and Narin and Bibliography 43 specifically van Raan deal with the methodology used in bibliometric program evaluation The special issue published by Research Evaluation on evaluation practices in Europe provides some interesting approaches to national experiences BOBE B Trends in the use of research and development output indicators in EC program evaluation Scientometrics Vol 21
43. categories Whereas output indicators are used to evaluate scientific output indicators linked to co author analysis and citation analysis are used to evaluate relevance Using bibliometrics in evaluating the efficiency of research management requires that research topics be identified specifically so as to detect possible synergies or pinpoint unwanted duplication Bibliometric techniques can be used to set up lists of actors and research topics providing administrators with an overview of the available resources and allowing for their use in restructuring an organisation on the basis of the structure of research 4 1 Impact measurements One of the problems raised by bibliometric evaluation is how to define and measure the quality of the output Two major issues have governed the bibliometric procedures used for the evaluation of scientific research i e the notion of quality and that of the impact of research In the beginning bibliometric evaluations attempted to measure the quality of research on the basis of citations It was felt that the number of citations was an indication of research quality The more often a document was cited the higher its quality The citation indicator was soon criticised In addition to its partial character critics had serious doubts about its true meaning Many researchers raised questions about the reasons why authors chose the publications they quoted This line of reasoning raised serious doubts about
44. ch evaluation in terms of performance National experiences are included with papers by Barbarie for Canada and Cozzens for the United States Kostoff is particularly interested in the evaluation of public research at the federal level The paper by Helander reviews the experience of northern countries Evaluation studies also deal with fields of research Examples are the paper by Frame on basic research and the contribution by Finkenstaedt on research in the humanities of particular interest since bibliometric studies dealing with the humanities and social sciences are rare indeed Gibbons and Georghiou provide an overview of evaluation methods including bibliometric methods In this context the paper by Moed compares bibliometric evaluation methods with the traditional technique of peer evaluation The two methods are often contrasted but they should in fact be seen as complementary A contribution by Vinkler deals with the delicate subject of evaluating individual performance BARBARIE A Evaluating federal R amp D in Canada in Barry Bozeman and Julia Melkers Eds Evaluating R amp D impacts Methods and practice Boston Kluwer Academic Publishers 1993 COZZENS S U S Research assessment Recent developments Scientometrics Vol 34 No 3 1995 pp 351 362 FINKENSTAEDT T Measuring research performance in the humanities Scientometrics Vol 19 No 5 6 1990 pp 409 417 FRAME J D Quantitative indicators for evalua
45. cience and technology Such interactions constitute the flow of knowledge The methods known as co word analysis and co citation analysis are also relational indicators They provide a picture of scientific activity based on the content of publications Such indicators help monitor changes in science and technology and identify emerging research topics and the relevant contributors Co citation analysis and co word analysis are rarely used 10 B Godin The state of science and technology indicators in the OECD countries Research paper Science and Technology Redesign Project Statistics Canada 1996 18 11 B Godin The state of science and technology indicators in the OECD countries Research paper Science and Technology Redesign Project Statistics Canada 1996 17 12 The Observatoire des Sciences et des Technologies database of Canadian authors as it now stands does not allow for the use of such indicators because of the lack of data on references and key words Bibliometrics Applied to Public Policy Methods and Examples 11 for policy purposes unlike descriptive indicators and co author analysis which are currently used in the description and evaluation of research 2 Analysis of scientific output A listing of a country s scientific publications is an indicator that can be used for the detailed analysis of scientific output Depending on the objectives of a given study this indicator is used to measure the relative weight of a c
46. cientometrics Vol 28 No 3 1993 pp 349 373 IRVINE J Evaluation of scientific institutions Lessons from a bibliometric study of UK technical universities in D Evered and S Harnett Eds The evaluation of scientific research Chichester UK John Wiley and Sons 1989 KRAUSKOPF M M IN S VERA and R ALBERTINI Assessment of a university s scientific capabilities and profile The case of the faculty of biological sciences of the Pontificia Universidad Catolica de Chile Scientometrics Vol 34 No 1 1995 pp 87 100 KRULL W The Max Planck experience of evaluation Scientometrics Vol 34 No 3 1995 pp 441 450 MARTIN B R and J E F SKEA Academic research performance indicator An Assessment of the possibilities Report to the Advisory Board for the Research Councils and the Economic and Social Research Council SPRU University of Sussex 1992 MOED H F et al The use of bibliometric data for the measurement of university research performance Research Policy 14 1985 pp 131 149 MOED H F and A F J van Raan Indicators of research performance Applications in university research policy in A F J van Raan et al Eds The handbook of quantitative studies of science and technology Amsterdam Elsevier Science Publishers B V 1988 pp 177 192 48 Bibliography NARIN F Measuring the research productivity of higher education institutions using bibliometric techniques OECD DSTI SPR 85
47. e different from the number of authors The ISI data simply reproduce the information provided by the authors themselves in their publications The information has value only when it is aggregated not when it is applied to individuals Another advantage of the ISI databases is that they alone contain the complete list of references cited by each document These references can be used to measure the impact of an article by calculating the number of citations for this article over a given period The great advantage of the ISI databases is that they provide the complete list of addresses given by the authors something that is not done by other current databases such as PASCAL FRANCIS MEDLINE and INSPEC which only include the address of the first author The ISI databases also provide other advantages in terms of representativeness According to Garfield the top 2 000 journals among the 3 400 that are covered represent about 85 of published articles and 95 of articles cited 8 This means that there is a significant concentration effect in the overall scientific literature worldwide This type of concentration guarantees the representativeness of the SCI database However Leclerc feels that the SCI database is only truly representative of mainstream scientific activity i e those scientific currents that feed and determine Western science A study based on a sampling of 2 500 periodicals surveyed in 1972 has shown that SCI provides excellent c
48. e position of scientific output and of researchers The goal is to provide a graphic representation of the structure of scientific output at a given moment in a two dimensional or three dimensional space As tools used to define specialities mapping and classification are based on co word analysis and co citation analysis The book by Braam is a comparative study of current mapping methods In the contribution by Engelsman and van Raan the method of key word mapping is applied to patents BRAAM R R Mapping of science Foci of intellectual interest in scientific research Leiden DSWO Press Science Studies Series No 4 1991 CALLON M J LAW and A RIP Mapping the dynamics of science and technology London Macmillan Press Ltd 1986 COZZENS S E and L LEYDESDORFF Journal systems as macro indicators of structural change in the sciences in A F J van RAAN et al Eds Science and technology in a policy context Select Proceedings of the Joint EC Leiden Conference on Science and Technology Indicators Leiden The Netherlands 23 25 October 1991 Leiden DSWO Press Science Studies Series No 8 1992 ENGELSMAN E C and A F J van Raan A patent based cartography of technology Research Policy Vol 23 pp 1 26 LEYDESDORFF L Various methods for the mapping of science Scientometrics Vol 11 Nos 5 6 1987 pp 295 324 LEYDESDORFF L and S COZZENS The delineation of specialties in terms of journals usi
49. ector which dominates the Canadian Economy The capacity to innovate resides in people and measures are being developed of the characteristics of people in those industries which lead science and technology activity In these same industries measures are being made of the creation and the loss of jobs as part of understanding the impact of technological change The federal government is a principal player in science and technology in which it invests over five billion dollars each year In the past it has been possible to say how much the federal government spends and where it spends it The current report Federal Scientific Activities Catalogue 88 204 released early in 1997 begins to show what the S amp T money is spent on with the new Socio Economic Objectives indicators As well as offering a basis for a public debate on the priorities of government spending all of this information will provide a context for reports of individual departments and agencies on performance measures which focus on outcomes at the level of individual projects By the final year of the Project in 1998 99 there will be enough information in place to report on the Canadian system on innovation and show the role of the federal government in that system As well there will be new measures in place which will provide a more complete and realistic picture of science and technology activity in Canada CONTACTS FOR MORE INFORMATION S amp T Redesign Project Direc
50. ences et des Technologies remains a tool capable of providing a general though qualitatively incomplete overview CONCLUSION This document has provided a general description of the bibliometric analysis of scientific and technological research The conceptual framework presented distinguishes relationships between three data aggregation levels i e micro meso and macro and three major variables years sectors and disciplines This type of conceptualisation helps differentiate bibliometric analyses according to the needs and interests of users The principal bibliometric indicators are presented and a distinction is made between descriptive indicators based on publication counts and relational indicators which highlight the links between researchers institutions and countries Publication counts and descriptive indicators form the basis of bibliometrics Possible applications are emphasised Descriptive indicators and macro analysis are described within the framework of the evaluation of national performance Co author analysis and various methods used for publication counts enable the analysis of international and national flow of knowledge The section on research evaluation introduces the issues of quality assessment and impact measurement The application of bibliometric methods to program evaluation and academic research evaluation are also discussed along with the method of convergent indicators Co citation analysis and co wo
51. enting a scientific field A bibliometric approach Scientometrics Vol 15 1989 pp 593 605 7 2 Indicators of technological innovation Narin and Carpenter and Coward and Franklin are the specialists of bibliometric analysis applied to patents This method is used to study technological innovation as well as the links between science and technology Computer Horizons have developed a method for evaluating the technological performance of a firm a technological sector or even a national technological activity Likewise Carpenter and Narin have developed the analysis of citations in patents as a means of measuring the links between scientific literature and technological development Coward and Franklin used co citation analysis to explore the links between science and technology Both methods are currently used to identify the areas of research having a strong technological 56 Bibliography potential and inversely those technologies that rely on basic research The paper by Pavitt looks at the problems and difficulties linked to the use of patents as an indicator of technological innovation ARCHIBUGI D Patenting as an indicator of technology innovation A review Science and Public Policy Vol 16 No 6 December 1992 BASBERG B Patents and the measurement of technological change A survey of the literature Research Policy Vol 16 1987 pp 131 141 CARPENTER M P and F NARIN Validation study Patent citations as indica
52. ers are available from the Science and Technology Section of Statistics Canada please contact Science and Technology Section Science and Technology Redesign Project Statistics Canada Ottawa Ontario K1A 0T6 Tel 613 951 6347 ST 97 01 A Compendium of Science and Technology Statistics February 1997 Price 75 00 ST 97 02 ST 97 03 ST 97 04 ST 97 05 ST 97 06 ST 97 07 ST 97 08 ST 97 09 ST 97 10 ST 97 11 ST 97 12 ST 97 13 Provincial Distribution of Federal Expenditures and Personnel on Science and Technology 1994 95 February 1997 Price 75 00 Scientific and Technological Activities of Provincial Governments 1989 90 to 1995 96 March 1997 Price 75 00 Federal Government Expenditures and Personnel on Activities in the Natural and Social Sciences 1987 88 to 1996 97 March 1997 Price 75 00 Transfers of Funds for Research and Development in Canadian Industry 1993 March 1997 Price 75 00 Estimation of Research and Development Expenditures in the Higher Education Sector 1995 96 August 1997 Price 75 00 Estimates of Canadian Research and Development Expenditures GERD Canada 1986 to 1997 and by Province 1986 to 1995 August 1997 Price 75 00 Federal Government Expenditures and Personnel on Activities in the Natural and Social Sciences 1988 89 to 1997 98 July 1997 Price 75 00 R amp D Tax Treatment in Canada A Provincial Comparison September 1997 Price 75 00 Pro
53. ew of the state of Canadian research in semiconductors and photonics La Physique au Canada July August 1996 pp 151 158 GAGNE J and M LECLEC Quebec s science indicators A survey Science and Public Policy Vol 19 No 5 October 1992 GEORGHIOU L Evaluation of research Evaluation of research in Canada Ad hoc Group on Scientific and University Research Directorate for Science Technology and Industry Paris OECD January 1986 GINGRAS Y Bibliometric analysis of funded research A feasibility study Report to the Program Evaluation Committee of NSERC Montr al Centre interuniversitaire de recherche sur la science et la technologie CIRST January 1996 GINGRAS Y C MEDAILLE La langue de publication des chercheurs qu b cois en sciences naturelles g nie et sciences biom dicales 1980 1988 Qu bec Minist re de l Enseignement sup rieur et de la Science Direction du d veloppement scientifique 1991 GINGRAS Y M LECLERC C MEDAILLE La recherche scientifique au Qu bec production et productivit de 1980 a 1988 Interface Vol 12 No 3 May June 1991 pp 25 32 52 Bibliography GODIN B Endangered Species Une nouvelle estimation de la part du francais dans les communications scientifiques Rapport de recherche pr sent au Conseil de la langue fran aise Montr al INRS Centre interuniversitaire de recherche sur la science et la technologie CIRST November 1995 GODIN B Profil des th
54. hout the world The compendium of research in Quebec represents a unique tool along with the bibliometric study by Leclerc on international scientific co operation in Quebec This is an excellent example of the use of bibliometrics for technological monitoring by a provincial government in Canada This section also contains national statistical works from France Observatoire des sciences et des techniques the United States National Science Foundation as well as the European Union BARRE R Ed Science et technologie Indicateurs 1996 Rapport de l Observatoire des Sciences et des Techniques Paris Economica 1996 BRAUN T W GL NZEL MACZELKA HAJNALKA and A SCHUBERT World science in the eighties National performances in publication output and citation impact 1985 1989 versus 1980 1984 Part II Life sciences engineering and mathematics Scientometrics Vol 31 No 1 1994 pp 3 30 BRAUN T W GLANZEL and H GRUPP The scientometric weight of 50 nations in 27 science areas 1989 1993 Part I All fields combined mathematics engineering chemistry and physics Scientometrics Vol 33 No 3 1995 pp 263 293 BRAUN T W GLANZEL and H GRUPP The scientometric weight of 50 nations in 27 science areas 1989 199 Part II Life sciences Scientometrics Vol 34 No 2 1995 pp 207 237 BRAUN T W GL NZEL and A SCHUBERT Scientometric indicators datafiles Scientometrics Vol 28 No 2 1993 pp 137 15
55. ibution to science technology and education Establish a relationship between the input and the output 2 Use the institutional approach Analysis is based on research units and not on individuals 3 Use the comparative method and apply it to analytical units that are similar 4 Combine various indicators which each reflect a different aspect of research performance Each indicator is partial and imperfect and reflects part of the relative importance of scientific contributions and part of the relevant social institutional psychological and other factors The principal indicators used are article output citation analysis and peer evaluation along with the number of researchers and research grants Source F Anderson New approaches to research policy using bibliometrics Quebec Conseil de la science et de la technologie February 1987 Ben Martin and John Irvine Assessing basic research Some partial indicators of scientific progress in radio astronomy Research Policy 12 1983 61 90 4 3 Evaluation of academic research Traditionally the evaluation procedure in universities has been based on peer review However bibliometric data are a good complement to traditional methods especially when the evaluation deals with research groups administrative units such as departments and faculties or even the institution as a whole The bibliometric evaluation of individuals however remains problematic and controversial
56. ics as well as empirical bibliometric studies References are listed according to the broad subdivisions of the guide itself providing readers with concrete information about the applications of bibliometrics The author has emphasised various national experiences including several Canadian studies 6 OECD Technology and the economy The key relationships Paris OCDE 1992 7 Niosi J P Saviotti B Bellon and M Crow 1993 National systems of innovation In search of a workable concept Technology and Society 15 207 227 Lundvall B A 1992 National systems of innovation Towards a theory of innovation and interactive learning Pinter London Nelson R R ed 1993 National innovation systems A comparative analysis New York Oxford University Press 8 Yoshiko Okubo Bibliometric Indicators and Analysis of Research Systems Methods and Examples Paris Organisation for Economic Cooperation and Development STI Working Papers 1997 1 1997 A BIBLIOMETRICS APPLIED TO PUBLIC POLICY METHODS AND EXAMPLES Scientometrics can be defined as the measurement of scientific and technical research activity Bibliometrics is a branch of scientometrics that focuses principally on the quantitative study of scientific publications for statistical purposes Bibliometric methods serve three main functions i e description evaluation and scientific and technological monitoring As a descriptive tool bibliometrics provides an account of
57. ie CIRST May 1995 GODIN B and M TREPANIER Les mat riaux composites matrice polym rique Rapport de recherche pr sent l Ordre des ing nieurs du Qu bec et au minist re de l Industrie du Commerce de la Science et de la Technologie Montr al INRS Centre interuniversitaire de recherche sur la science et la technologie CIRST May 1994 Bibliography 53 INHABER H The leading edge of science in Canada Research Policy Vol 7 1978 pp 88 98 LECLERC M La coop ration scientifique du Canada avec les autres pays membres de l ALENA et certains pays d Amerique latine 1980 1990 Rapport pr par pour Industrie Canada Bureau du conseiller scientifique principal Qu bec April 1994 LECLERC M La coop ration scientifique internationale l exemple du Canada La recherche Vol 24 n 252 March 1993 pp 350 357 LECLERC M and J GAGNE La coop ration scientifique internationale du Qu bec Nouvelles de la science et des technologies Vol 9 n 2 1991 pp 133 136 LECLERC M J F MIQUEL N NARVAEZ L FRIGOLETTO and Y OKUBO La coop ration scientifique France Qu bec analyse des cosignatures Interface vol 12 n 5 septembre octobre 1991 pp 19 23 LECLERC Michel Y OKUBO L FRIGOLETTO and J F MIQUEL Scientific co operation between Canada and the European Community Science and Public Policy Vol 19 No 1 February 1992 pp 15 24 LEYDESDORFF L and E GAUTHIER The eva
58. ies Scientometrics Vol 36 No 3 1996 pp 421 434 MOED H F W J M BURGER J G FRANKFORT and A F J van RAAN La bibliom trie et son utilit pour l orientation g n rale des recherches universitaires Revue internationale de gestion des tablissements d enseignement sup rieur Vol 9 1985 pp 210 220 MOED H F R E de BRUIN A J NEDERHOF and R J W TIJSSEN International scientific cooperation and awareness within the European Community problems and perspectives Scientometrics Vol 21 1991 pp 291 311 MOED H F and F T HESSELINK The publication output and impact of academic chemistry research in the Netherlands during the 1980s Bibliometric analyses and policy implications Research Policy Vol 25 1997 p 819 NAGPAUL P S and L SHARMA Science in the eighties A typology of countries based on inter field priorities Scientometrics Vol 34 No 2 1995 pp 263 283 Van RAAN A F J Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises Scientometrics Vol 36 No 3 1996 pp 397 420 6 Bibliometric studies of Canada There are several published bibliometric studies dealing with specific aspects of scientific activity in Canada These studies cover a broad range of subjects reflecting the existence of a group of active bibliometric specialists In addition to scientific publications there are many research reports The following selection
59. ington D C U S Government Printing Office 1996 SCHUBERT A and T BRAUN International collaboration in the sciences 1981 1985 Scientometrics Vol 19 No 1 2 1990 pp 3 10 SCHUBERT A W GL NZEL and T BRAUN Scientometrics data files A comprehensive set of indicators on 2 649 journals and 96 countries in all major fields and subfields 1981 1985 Scientometrics Vol 16 No 1 6 1989 pp 3 478 UNESCO World science report 1993 Paris UNESCO 1993 36 Bibliography UNESCO Statistics on science and technology Paris UNESCO October 1991 3 Indicators 3 1 Indicators in general The contributions listed in this section deal with bibliometric indicators used as analytical tools as opposed to describing their use in a specific context They describe the validity and use of various indicators in general They represent a source of information on scientometric indicators as a whole The paper by Edge is a review of the principal bibliometric indicators It also talks about limits and weaknesses The documents published by OECD provide essential information about the utilisation of indicators They establish statistical standards for all OECD countries Other documents dealing with national experiences have been included to provide a broader account of indicators in general Thus the paper by Zitt and Teixeira discusses the specific experience of the Observatoire des Sciences et des Techniques This institute has been pr
60. iography KUHLMANN S German government departments experience of RT amp D programme evaluation and methodology Scientometrics Vol 34 No 3 1995 pp 461 471 LARADO P Structural effects of EC RT amp D programmes Scientometrics Vol 34 No 3 1995 pp 473 487 LEYDESDORFF L and P van der SCHAAR The use of scientometric methods for evaluating national research programs Science and Technology Studies Vol 5 No 1 1987 pp 22 31 MOED H F The use of bibliometric indicators for the assessment of research performance in the natural and life sciences Leiden DSWO Press Science Studies Series No 2 1989 MULLINS N C Evaluating research programs Measurement and data sources Science and Public Policy Vol 14 No 2 April 1987 pp 91 98 NARIN F Bibliometric techniques in the evaluation of research programs Science and Public Policy Vol 14 No 2 April 1987 pp 99 106 NEDERHOF A J A bibliometric assessment of research council grants in linguistics Research Evaluation Vol 6 No 1 April 1996 Van RAAN A F J Advanced bibliometric methods to assess research performance and scientific development Basic principles and recent practical applications Research Evaluation Vol 3 No 3 December 1993 WALSCHE G Research and development trends Criteria for assessment Science and Public Policy Vol 19 No 2 1992 pp 75 88 4 3 Evaluating national performance This sectio
61. is by no means exhaustive ANDERSON F and R DALP A comparison of alternative data sources to profile national research communities The case of Canadian polymer research Montr al Centre interuniversitaire de recherche sur la science et la technologie CIRST Notes de recherche 92 05 1992 ANDERSON F and R DALP A profile of Canadian coal and petroleum research communities Scientometrics Vol 25 No 3 1992 pp 447 463 DALP R International activities of public laboratories in Canada Technology in Society Vol 19 No 2 1997 pp 127 143 Bibliography 51 DALPE R and ANDERSON F National priorities in academic research Strategic research and contracts in renewable energies Research Policy Vol 24 1995 pp 563 581 DALPE R E GAUTHIER M PARENT and M PRUD HOMME Les activit s internationales des laboratoires publics au Qu bec Qu bec Conseil de la science et de la technologie November 1993 DALPE R and Y GINGRAS Recherche universitaire et priorit s nationales l effet du financement public sur la recherche en nergie solaire au Canada Canadian Journal of Higher Education Vol 20 1990 pp 27 44 DALPE R and B LONGPRE The state of Canadian research in physics and electrical engineering Report to the NRCC Montr al Centre interuniversitaire de recherche sur la science et la technologie CIRST August 1995 DAWSON P R DALPE B LONGPRE and C CARON A bibliometric vi
62. is no need to wait a long time for the data it costs less than a citation count and finally sources of error are minimised Furthermore for a given discipline there is an excellent correlation between impact factors and the relative significance of journals as evaluated by researchers Since there is a specific impact factor for each discipline and even each speciality comparative studies are only carried out within the relevant disciplines and specialities 23 P R McAllister R Anderson and F Narin Comparaison of Peer and Citation Assessment of the Influence of Scientific Journals Journal of the American Society for Information Science Vol 31 1980 pp 147 152 J T Wallimack and K D Sedig Quality of Research Measured by Citation Method and by Peer Review A Comparaison EEE Transactions on Engineering Management Vol 1 Em 33 November 1986 Bibliometrics Applied to Public Policy Methods and Examples 21 Example 7 Program evaluation the National Institutes of Health The grant programs of the American institutes linked to the National Institutes of Health NIH have been evaluated using the bibliometric method since 1973 Computer Horizons Incorporated CHI have worked under contract with NIH to develop a bibliometric database to evaluate the impact of NIH in terms of publications and citations The evaluation report entitled An Analysis of Research Publications Supported by NIH 1973 80 makes the following observ
63. is part of the information contained in a document Institutional sector Generally speaking science and technology policies recognise three main locations where research is carried out universities industry and public laboratories However given the characteristics of the database of the Observatoire des Sciences et des Technologies which includes the natural sciences engineering biology and medicine the humanities as well as the arts new sectors of activity had to be added in order to better reflect all the people involved in research in Canada Such sectors correspond to types of institutions that can be grouped together in terms of analytical requirements There are 16 types of institutions in the database The definition of a sector of activity therefore includes one or several types of institution depending on the specific needs of analysis Various groupings are used whenever the database is consulted APPENDIX II Specification of the database of the Observatoire des Sciences et des Technologies A Information sources The database of Observatoire des Sciences et des Technologies was established using data from Science Citation Index SCD Social Science Citation Index SSCI and Art amp Humanities Citation Index AHCD It was established in four major steps information gathering information processing data standardisation and codification The last two operations are what distinguish the database from other bibliogr
64. l provide interested readers with an overview GARFIELD E Citation indexing Its theory and application in science technology and humanities New York Wiley 1979 GARFIELD E and A WELLJAMS DOROF Citations data Their use as quantitative indicators for S amp T evaluation and policy making Science and Public Policy Vol 19 No 5 October 1992 NARIN F Evaluative bibliometrics The use of publication and citation analysis in the evaluation of scientific activity Contract NSF C 627 National Science Foundation March 31 1976 Monograph 456 NTIS Accession PB252339 AS 58 Bibliography MILMAN B L and Y A GAVRILOVA Analysis of citation and co citation in chemical engineering Scientometrics Vol 27 No 1 1993 pp 53 74 SMALL H and SWEENEY E Clustering the Science Citation Index using co citations Part I A comparison of methods Scientometrics Vol 7 1985 pp 391 409 SMALL H E SWEENEY and GREENLEE E Clustering the Science Citation Index using co citations Part II Mapping science Scientometrics Vol 8 1985 pp 331 340 WADE N Citation analysis A new tool for science administrators Science Vol 188 1975 pp 429 432 WALLMARK J T and K G SEDIG Quality of research measured by citation method and by peer review A comparison IEEE Transactions on Engineering Management Vol 33 No 4 1986 7 4 Co word analysis Co word analysis and co citation analysis are both used t
65. lication on the basis of the address Publications by authors in other countries or in the rest of Canada as well as publications for which no sector of activity could be identified were assigned to the category others Collaboration within a sector corresponds to the co authorship of a publication by authors in the same sector Collaboration between sectors corresponds to publications whose authors are from different sectors The author concluded that the university sector remains the most prolific with 60 of all publications in 1990 and the share of publications in other sectors is rapidly increasing at the expense of the university sector Nevertheless universities remain the most frequent partners in collaborative efforts between sectors in terms of co authorships Source B Godin Besides Academic Publications Which Sectors Compete or Are There Competitors Scientometrics Vol 33 No 1 1995 4 Research evaluation Evaluation strategic management and futures analysis have become frequent activities within the framework of scientific and technological policy In this respect bibliometric methods are aimed principally at the decision making process They provide quantitative data to supplement the qualitative evaluation process Although bibliometric indicators were developed in the post war period public policy makers have become interested in such indicators fairly recently As public policy makers came to see scie
66. ll known theoretical input output model Investments input are applied to various scientific and technical activities that lead potentially to both knowledge and applications output Until recently most of the efforts reflections studies and indicators of OECD and Western countries were focused on input This situation can be explained on historical grounds In fact input indicators are linked directly to the emerging scientific and technological policies of the 1960s and 1970s The chief goal at that time was to take part in research and development funding Input indicators helped determine where funds were invested and where there was less research It was hoped that this would provide governments with tools indicating the best possible use of public funds In view of the complexities of science and technology many of the indicators of the past have become less inadequate as a measurement of reality Furthermore during the 1980s governments became more and more interested in innovation and the impact of research This is in fact the focus of recent federal strategies for science and technology in Canada In this context OECD designed the Oslo Manual which deals with aspects of research and development directly linked to innovation i e the creation of new products and processes OECD also took up a new challenge in 1992 with the TEP Program Technology Economy l Benoit Godin Yves Gingras and Louis Davignon Knowledge flo
67. luation of national performance in selected priority areas using scientometric method Research Policy Vol 25 1996 pp 431 450 MACAULEY James B An indicator of excellence in Canadian science Ottawa Statistics Canada Catalogue 88 501E 1985 MARCHESSAULT R R DALPE and E GAUTHIER Evaluation of Canadian research in metals and alloys Report to the Royal Society of Canada 1992 REES POTTER Lorna K Identification of areas of social science research in Canada A bibliometric analysis Proceedings of the American Society for Information Science 17 1980 pp 282 284 TREPANIER M and B GODIN Les mat riaux composites matrice polym rique les bior acteurs les technologies de reconnaissance de la voix et de l image trois rapports de recherche pr sent s l Ordre des ing nieurs du Qu bec et le minist re de l Industrie du Commerce de la Science et de la Technologie du Qu bec 1994 54 Bibliography SMITH W A Evaluating research technology and development in Canadian industry Meeting the challenges of industrial innovation Scientometrics Vol 34 No 3 1995 pp 527 539 SYLVAIN C Canadian research activity in aquaculture A bibliometric analysis Scientometrics Vol 27 No 3 1993 pp 295 316 7 Supplementary reading 7 1 Classification and mapping techniques This section contains contributions on various methods of graphic representation Mapping is the preparation of maps showing th
68. matiques de la recherche qu b coise application au g nie Rapport de recherche pr sent au Fonds FCAR Montr al INRS Centre interuniversitaire de recherche sur la science et la technologie CIRST June 1995 GODIN B Research and the practice of publication in industries Research Policy Vol 25 1997 pp 587 606 GODIN B Profil bibliom trique de la recherche financ e en sciences naturelles g nie et sciences biom dicales Rapport de recherche pr sent au Fonds FCAR Montr al INRS Centre interuniversitaire de recherche sur la science et la technologie CIRST April 1997 GODIN B R S BARKER and M LANDRY Besides academic publications Which sectors compete or are there competitors Scientometrics 33 1 1995 pp 3 12 GODIN B and M P IPPERSIEL Scientific collaboration at the regional level The case of a small country Scientometrics 36 1 1996 pp 59 68 GODIN B and R LANDRY L avenir de la collaboration scientifique au Qu bec une analyse bas e sur la convergence d indicateurs Rapport de recherche pr sent au fonds FCAR Montr al INRS Centre interuniversitaire de recherche sur la science et la technologie CIRST January 1995 GODIN B and C LIMOGES Les revues scientifiques qu b coises une valuation du programme de soutien aux revues du fonds FCAR Rapport de recherche pr sent au fonds FCAR Montr al INRS Centre interuniversitaire de recherche sur la science et la technolog
69. ment 4 Industry private enterprise industry 5 Other library college school church art gallery museum orchestra and music theatre other Likewise the data on cities provinces and countries found in the addresses can be used to analyse scientific activity in a given administrative region A 6 APPENDIX II Specification of the database of the Observatoire des Sciences et des Technologies B Database structure The documentation on a database is not complete unless the structure of the information it contains has been described in detail This section provides a general overview of the database of the Observatoire des Sciences et des Technologies It includes all three files and their contents The database was given a relational structure The operating principle is fairly simple The information is classified by segments in several specialised files Data reconstruction requires that a link be established between the different files This link is provided by a common key for all files Thus the database includes three files The article file contains complete bibliographic data for all the documents This central file is complemented by two other files the author file and the address file A document number common to all files is the key to data reconstruction for a given document Each file operates individually but together they form a whole The diagram in Figure A shows how the information contained in the database is
70. metric study on the state of international nanotechnological research The goal was to provide information on the development of research and technology within the framework of its nanotechnology program Two aspects of research were targeted a description of the major orientations characterising pure and applied research and an analysis of industrial research activities in nanotechnology and any links with pure and applied research Bibliometric data were gathered from three sources the INSPEC database which covers among other things physics and electrical engineering the chemical section of the SCI database along with US Patent Search Claims amp Abstracts The study was carried out using a list of key words defining the field of nanotechnology The list was provided by experts at the National Research Council A database of publications containing any one of the key words either in the title or in the abstract was established for the period from 1989 to 1996 A similar database was set up for patents The report contained the following information e frequency lists by key word for scientific publications and patents e the structure of the field illustrated by graphs bringing together key words on the basis of their co occurrence graphs illustrating research fields associated with Canadian research institutions a description of publication and patent trends during the period in question the world distribution of publications and patents
71. n lists sources on the utilisation of bibliometric tools to evaluate national performance Some of them discuss the method while others e g Callon and Leydesdorff provide an illustration of bibliometric evaluation The papers by Leydesdorff and Gauthier and by Sylvain describe the evaluation of a specific field of scientific activity materials research and biotechnology on the one hand and aquaculture on the other The paper by Leydesdorff and Gauthier is based on a comparative study of strategic research programs in Canada and the Bibliography 45 Netherlands The contribution by Gl nzel contains a bibliometric evaluation of research in the social sciences Few bibliometric studies have dealt with this field of scientific research BRAUN T W GL NZEL and A SCHUBERT An alternative quantitative approach to the assessment of national performance in basic research in D EVERED and S HARNETT eds The evaluation of scientific research Chichester UK John Wiley amp Sons 1989 CALLON M and L LEYDESDORFF La recherche frangaise est elle en bonne sant La recherche Vol 18 No 186 1987 pp 412 419 GLANZEL W A bibliometric approach to social sciences National research performances in 6 selected social science areas 1990 1992 Scientometrics Vol 35 No 3 March 1996 pp 291 307 KYVIK S and I MARHEIM LARSEN International contact and research performance Scientometrics Vol 29 No 1 1994 pp 137 159
72. n results It is therefore very important for users of a scientific publication database to take this kind of limitation into consideration In summary the representativeness of bibliometric data depends on which specialised indices are used as a source The SCI database has more advantages than disadvantages In the natural sciences engineering and the biomedical sciences articles in English make up the bulk of the scientific output Furthermore scientific journals indexed by SCI account for most of the citations and are thus representative of the bulk of the scientific output On the other hand SCI shows significant bias in terms of coverage by country publications in foreign languages other than English as well as coverage of publications in the applied sciences In spite of these shortcomings SCI remains a valid bibliographic source The SSCI and AHCI databases on the other hand merit a more qualified description The dynamics of the social sciences the humanities and the arts are different from those of the disciplines covered by SCI In fact a significant share of scientific output is published in the form of monographs Moreover since these disciplines are of a more contextual nature researchers publish more often in their own language As a result any bias in favour of English would underestimate a significant portion of the scientific output in these disciplines That is why SSCI and AHCI are considered to be of little value unless
73. nce as a 14 Sectors of activity include different types of institutions The types of institutions and the ways in which they are classified are explained in detail in the glossary of Appendix II 15 A detailed list of sectors with specifications is found in Appendix I of this guide 18 Bibliometrics Applied to Public Policy Methods and Examples cultural asset then as a public asset and then as an economic asset 6 the need to measure the results of science became more and more pressing As a result the use of bibliometric methods of evaluation became more widespread as an information tool for public policy makers Evaluation procedures can be applied to all three levels of analysis identified above i e at the micro level researchers the meso level research programs and the macro level provincial and national research systems Callon et al described three major categories of issues associated with research evaluation The first category is related to evaluations of the volume of scientific output The second category deals with the relevance of research in terms of its impact on the development of knowledge or on society and the economy Evaluations of relevance are aimed at assessing the choice of research topics and of grantees as well as their ripple effects on the dynamics of research The third category deals with the efficiency of research management Bibliometric evaluation procedures are directly linked to the first two
74. nd J F Miquel Scientific co operation between Canada and the European Community Science and Public Policy Vol 19 No 1 February 1992 p 16 28 Bibliometrics Applied to Public Policy Methods and Examples Garfield has shown that the probability of being cited for French and Quebec researchers publishing in French was between 30 and 50 lower than for anglophone researchers These observations would tend to confirm that francophone researchers including Quebec researchers are underrepresented in the SCI database Nevertheless this trend is countered by the fact that a vast majority 95 of francophone researchers in Quebec universities is now publishing in English 4 The types of documents that are compiled also represent an important limit for the ISI databases In spite of its systematic coverage of periodicals gt ISI does not compile documents that are distributed outside existing dissemination channels Such documents known as grey literature include theses internal reports research notes patents and communications that have not been published Monographs are not surveyed in spite of the fact that they represent a significant part of the scholarly output in the humanities and social sciences Certain reservations should therefore apply whenever bibliographic data from the ISI databases are used for evaluation purposes The smaller the analytical unit the higher the risk that the choice of data will affect the evaluatio
75. ng the dynamic journal set of the SCI Scientometrics Vol 26 1993 pp 133 154 Bibliography 55 Van RAAN A F J and R J W TIJSSEN Numerical methods for information on aspects of science scientometrics analysis and mapping in Oppenheim C J M Griffiths and C L Citroen Eds Perspective in information management Vol 2 London Butterworths 1990 REES POTTER Lorna K Identification of areas of social science research in Canada A bibliometric analysis Proceedings of the American Society for Information Science 17 1980 pp 282 284 ROTHMAN Harry Science mapping for strategic planning in Michael Gibbons Philip Gummett and B M Udgaonkar Eds Science and technology policy in the 1980s and beyond London and New York Longman s 1984 pp 99 116 SMALL H and E GARFIELD The geography of science Disciplinary and national mapping Journal of Information Science Vol 11 1985 pp 147 159 SMALL H and B C GRIFFITH The structure of scientific literature Part II Identifying and graphing specialties Science Studies Vol 4 1974 pp 339 365 TIJSSEN R J W Cartography of science Scientometric mapping with multidimensional scaling methods Leiden DSWO Press Science Studies Series No 5 1992 TIJSSEN R J W and A F J van Raan Mapping changes in science and technology in R N Kostoff Ed Special issue on research impact assessment Evaluation Review 18 1 1994 TODOROV R Repres
76. ng paper Science and Technology Redesign Project Statistics Canada 1998 B BIBLIOGRAPHY 1 General sources on bibliometrics This section contains sources that deal with scientometrics and bibliometrics These are in fact manuals for readers who wish to know more about the subject They provide a good overview of methods and indicators as well as of the problems raised by the choice and use of information databases The document prepared by Anderson provides a good introduction its approach is firmly based on applications to research policy Although less recent it remains significant because it is very well documented Courtial as well as Callon Courtial and Penan provide a general introduction to scientometrics They give a good overview of co word analysis These works are well structured and they provide a wealth of information However they say little about bibliometric studies by non French researchers Other works in this bibliography complete the picture concerning American Dutch and British contributions Generally speaking the principal journals in the field of applied bibliometrics are Scientometrics Research Policy Science and Public Policy and Research Evaluation ANDERSON Frances New approaches to research policy using bibliometrics Qu amp bec Conseil de la science et de la technologie document interne n 4 February 1987 CALLON Michel Jean Pierre COURTIAL and Herv PENAN La scientometrie Paris Presses
77. niversity departments Some problems and partial solutions in A F J van RAAN et al Eds Science and technology in a policy context Select Proceedings of the Joint EC Leiden Conference on Science and Technology Indicators Leiden The Netherlands 23 25 October 1991 Leiden DSWO Press Science Studies Series No 8 1992 pp 106 121 ZACHOS G Research output evaluation of 2 university departments in Greece with the use of bibliometric indicators Scientometrics Vol 21 No 2 1991 pp 195 221 Bibliography 49 5 Scientific and technological monitoring The various contributions brought together in this section are an illustration of the broad potential of bibliometrics Relevant bibliometric data can be used to provide the information needed by decision makers dealing with a specific subject Such information may be used by public policy makers as well as by business leaders Thus bibliometrics might provide information about the technological dynamics of a city Dou ef al or of an industry Koening Bibliometrics can also be used to guide research in accordance with managerial considerations These are issues of strategic positioning as is shown by Miller and Manseau with respect to research and development laboratories Moed and colleagues on the subject of university research or Hicks Martin and Irvine on the topic of applied research Moreover the results of bibliometric studies help provide information for the peer
78. nowledge flow in the natural sciences and engineering The second The Use of Bibliometric Data to Measure Scientific Production in the Arts Humanities and Social Sciences A Methodological Note examines the issues involved in the use of bibliometrics for the social sciences arts and humanities Both of these working papers are authored by Benoit Godin Yves Gingras and Louis Davignon of the Observatoire des Sciences et des Technologies The objective of the Information System for Science and Technology Project at Statistics Canada is to develop useful indicators of activity and a framework to tie them together into a coherent picture of science and technology in Canada Bibliometric indicators of science and technology provide an important contribution to the understanding of the production of science and technology as measured by the production of scientific publications and of knowledge flows within the science and technology system as measured by co authorships in scientific publications Bibliometric indicators can shed light on science and technology production and knowledge flow at the international national provincial sub regional municipal and institutional levels and thus constitute a critical component of the information system on science and technology for Canada The bibliometric project supported by Statistics Canada created a Canadian database of bibliometric information This involved the cleaning of the 1995 data from the selec
79. o 2 The Effect of Country of Control on Industrial Research and Development R amp D Performance in Canada 1993 No 3 The Provincial Research Organizations 1995 No 4 Federal Government Expenditures on Scientific Activities 1997 98 No 5 Industrial Research and Development 1993 to 1997 No 6 Software Research and Development R amp D in Canadian Industry 1995 No 7 Distribution of Federal Expenditures on Science and Technology by Province and Territories 1995 96 No 8 Total Spending on Research and Development in Canada 1986 to 1997 and Provinces 1986 to 1995 No 9 Estimation of Research and Development Expenditures in the Higher Education Sector 1995 1996 No 10 Research and Development R amp D Personnel in Canada 1986 to 1995 No 11 Biotechnology Research and Development R amp D in Canadian Industry 1995 No 12 Research and Development R amp D Expenditures for Environmental Protection in Canadian Industry 1995 No 13 Research and Development R amp D Expenditures of Private Non Profit PNP Organizations 1996 Volume 22 No 1 The Provincial Research Organizations 1996 No 2 Federal Government Expenditures on Scientific Activities 1998 99 No 3 Federal Government Personnel Engaged in Scientific and Technological S amp T Activities 1989 90 to 1998 99 No 4 Biotechnology Scientific Activities in Selected Federal government Departments and Agencies 1997 98 WORKING PAPERS 1997 These working pap
80. o identify the dynamics of research areas In both cases results are most often presented in the form of maps The first three papers in this list use both co word analysis and co citation analysis The last four papers use only co word analysis De BRUIN R E and H F MOED Delimitation of scientific subfields using cognitive words from corporate addresses in scientific publications Scientometrics Vol 26 No 1 1993 pp 65 80 CAMBROSIO A C LIMOGES J P COURTIAL and F LAVILLE Historical scientometrics Mapping over 70 years of biological safety research with co word analysis Scientometrics Vol 27 No 2 1993 pp 119 143 COURTIAL J P A co word analysis of scientometrics Scientometrics Vol 31 No 3 1994 pp 251 260 COURTIAL J P and J LAW A co word study of artificial intelligence Notes amp Letters Social Studies of Science Vol 19 No 2 pp 301 311 Bibliography 59 HINZE S Bibliographical cartography of an emerging interdisciplinary discipline The case of bioelectronics Scientometrics Vol 29 No 3 1994 pp 353 376 Van RAAN A F J and R J W TIJSSEN The neural net of neural network research An exercise in bibliometric mapping Scientometrics Vol 26 No 1 1993 pp 169 192 TIJSSEN R J W and A F J van Raan Mapping co word structures A comparison of multidimensional scaling and Leximappe Scientometrics Vol 15 No 3 4 1989 pp 283 295 ZITT M and BASSECOU
81. od does not take into account the contribution of the other authors It is based on the erroneous hypothesis that the first author is the most important one Moreover the link between the first author and the first address cannot be identified with certainty A third method consists in assigning the publication to each of the contributing countries Thus an article by co authors from different countries will be counted for each of those countries This means that the total number of publications will exceed the actual number of publications This method does Bibliometrics Applied to Public Policy Methods and Examples 15 not take into account the relative contribution of the authors It emphasises the participation of each country to the production of knowledge Example 3 International scientific co operation the example of Canada A study was carried out to determine Canada s profile in terms of international scientific collaboration The study showed that Canada is ranked last among the eight leading OECD countries in terms of the number of international patents the number of scientists and engineers in relation to the labour force the number of high technology industries having a positive trade balance and the amount of research and development financed by industry Between 1981 and 1986 Canada was ranked seventh among major countries producing scientific publications with 4 3 of all publications surveyed by the Institute for Scientifi
82. on it is more difficult to draw a representative picture of scientific activity in these fields using existing databases That is why the SSCI and AHCI databases cannot maintain that they provide truly representative coverage of Canadian publications This is even truer for documents written in languages other than English Table C Comparing two bibliometric databases Share of principal countries 1990 1992 SCI PASCAL N N United States 480 626 36 5 498 789 32 5 Japan 115 555 8 8 124 676 8 1 Federal Republic of Germany 98 664 7 5 98 723 6 4 France 76 652 5 8 112 213 7 3 United Kingdom 9 560 7 3 114 202 7 5 Canada 64 528 4 9 61 469 4 0 Quebec 14 237 1 1 12 592 0 8 Others 386 809 29 3 522 622 34 1 Total 1 318 440 100 1 532 694 100 Source Benoit Godin Profil bibliom trique de la recherche financ e en sciences naturelles g nie et sciences biom dicales Research Report Submitted to the FCAR Fund January 1997 At the end of the 1980s a study by Garfield showed that the number of francophone periodicals surveyed by SCI was gradually decreasing In 1985 SCI was surveying 46 Canadian periodicals as compared to 41 in 1988 Of these there were no Quebec periodicals in French Another study by 32 E Garfield French research citation analysis indicates trends are more than just a slip of the tongue Current Contents 23 1988 p 9 Cited in Leclerc Michel Y Okubo L Frigoletto a
83. onal performance seins 4 4 Evaluating publiGnstitutionss ss eds in ts Ds u a 5 Scientific and technological monitoring sise 49 6 Bibliometric studies of Ca ada micro iii 50 TESUPplementa y TEAM 54 7 1 Classification and mapping techniques ss 54 7 2 Indicators of technological innovation ses 55 7 3 Citation and co citation analysis 2525055525 Deore ea 57 LACH WOrd analysis 0 nn AU ae abe N nee 58 APPENDEX T Glossary ooo a ia A 1 APPENDIX II Specification of the database of the Observatoire des Sciences et des Technologies A 3 As Information SOurcessnna ib A 3 B Dat base structure A a ia dada sie te ete etait ea d s A 6 Figure A How a relational database operates ss A 6 Figure B Sample entry of the article file ss A 7 Figure C Sample entry of the author file ss A 8 Figure D Sample entry of the address file A 9 APPENDIX IH Classification of scientific disciplines oonocnnncnnonaconnnconnnoonncnoncnnnnconncnnn nc nnn conan cnn n crac cnn nccnnnccnnno A 11 APPENDIX IV Classification of disciplines in humanities social sciences arts and literature A 13 ACKNOWLEDGEMENTS I wish to extend special thanks to Yves Gingras and Benoit Godin of the Centre interuniversitaire de recherche sur la science et la technologie CIRST for their comments on the drafts of this document I also wish to thank Pierre Di Campo documentalist for his help in
84. ountry the output of researchers or the dynamics of a scientific field or an institutional sector The output of one country can then be compared to that of other countries in a competitive or comparable situation Comparisons between countries can also be based on disciplines Example shows three different aspects of the Canadian scientific output 12 Bibliometrics Applied to Public Policy Methods and Examples Example 1 Three aspects of Canadian scientific output 1995 Canadian share of world publications 1995 Others 32 Italy 4 Canada 4 inni inati Provincial share of publications 1995 Germany France j O 2 dl PEL NB_ Nfld 0 1 1 NWT 8 Yukon 0 Share of publications by sector 1995 Provincial govt 2 Business 4 Federal govt 11 Hospitals 15 University 65 Source Observatoire des sciences et des technologies CIRST March 1998 Bibliometrics Applied to Public Policy Methods and Examples 13 In an effort to measure aspects of national output a number of countries publish statistics on scientific publications by discipline A specialisation index has been developed to identify for a given country those disciplines that are over represented or under represented in terms of world averages for each sector This index is the ratio of the percentage of a country s publications in a given discipline to the percentage of publications in that discipline at the world level If the
85. overage for the United States and the United Kingdom good coverage for France and Germany and under coverage for publications from Japan the Soviet Union as well as other countries Moreover in terms of disciplines the coverage was best for research in medicine biomedical research and the physical sciences such as chemistry physics and mathematics On the other hand SCI provided less good coverage for engineering and technology in general as well as for the earth sciences The same reservations are expressed by OECD in its 1997 document Such considerations raise the issue of the inclusion of Canadian authors in the ISI databases The simple comparison shown in the TABLE C responds to this issue There is a slight variation in the coverage of scientific output for various countries depending on whether the ISI databases SCI SSCI AHCD or those of CNRS PASCAL FRANCIS are used Generally speaking however the ranking 28 These statistics apply to the SCI database CD ROM 1994 edition cf Eugene Garfield The Significant Scientific Literature Appears in a Small Core of Journals The Scientist September 2 1996 A few years previously Garfield had maintained the same proposition with very similar data Cf E Garfield How ISI selects journals for coverage quantitative and qualitative considerations Current Contents 22 1990 pp 5 13 29 M Leclerc Mesurer la science Dynamiques et mesures de la coop ration scientifique inte
86. oviding scientific and technical monitoring for years throughout France and it has developed a number of indicators BARRE R The European perspective on S amp T indicators Scientometrics Vol 38 No 1 1997 pp 71 86 CALLON M Les indicateurs des sciences et des techniques Recherche et Technologie n 1 January March 1986 p 42 EDGE D Quantitative measures of communication in science A critical review History of Science Vol 17 1979 pp 102 134 GALANTE E and C SALA R amp D evaluation at the Italian National Research Council The agricultural indicators Scientometrics Vol 36 No 2 pp 223 236 GILBERT G N Measuring the growth of science Scientometrics Vol 1 No 1 1978 pp 9 34 GLANZEL W The need for standards in bibliometric research and technology Scientometrics Vol 35 No 2 1996 pp 167 176 GRUPP H On the supplementary functions of science and technology indicators Scientometrics Vol 19 No 5 6 1990 pp 447 472 Bibliography 37 LECLERC M and Y GINGRAS Les indicateurs du financement priv de la R D universitaire au Qu bec critique de la m thode The Canadian Journal of Higher Education La revue canadienne d enseignement sup rieur Vol 23 No 1 1993 pp 74 107 Van der MEULEN B J R The use of S amp T indicators in science policy Dutch experience and theoretical perspectives from policy analysis Scientometrics Vol 38 No 1 1997 pp
87. perimental psychol Fertility Gastroenterology General psychology Genrl amp internal med Geriatrics Hematology Hygiene amp publ hlth Immunology Misc clinical med Misc psychology Nephrology Neurol amp neurosurg Obstetrics amp gynecol Ophthalmology Orthopedics Otorhinolaryngology Pathology Pediatrics Pharmacology Pharmacy Psychiatry Radiology amp nucl med Respiratory system Surgery Tropical medicine Urology Veterinary medicine Discipline Earth and space astronmy amp astrophys earth amp plantry sci environmental sci geography geology meteorol amp atmos sci oceanography amp limno Discipline Engineering amp techn aerospace technology chemical engineering civil engineering computers electr eng amp elctron general engineering industrial engineer library amp info sci materials science mechanical engineer metals amp metallurgy misc eng amp technol nuclear technology op res amp managmt sci Discipline mathematics applied mathematics general mathematics misc mathematics probablty amp statist Discipline Physics acoustics applied physics chemical physics fluids amp plasmas general physics miscellaneous phys nucl amp particle phys optics solid state physics APPENDIX IV Classification of disciplines in humanities social sciences arts and literature Discipline Anthropology Anthropology Ethnology Folklore Discipline Archeology Archeology Di
88. ple entry of the article file Be Ed Dots view Customize Gobas Options Window Help cll alol alle all el ci BIE el English An Ir 1 Dependert Pathway of 376 English 6 The Variable Effect of Clouds on 376 6539 ais 417 English Size and Structure of the 376 6539 424 427 English Eistinguishable Functions for 6539 244 47 English Saquastration of the 376 637 240 2 English Lalludina Gradiert of 376 5 43 English Identification and Inhibition of ihe 376 6535 52 Eb English Arte Failure of Bionc sland Formation 376 70 English Role of the FL 1 Receptor 375 E534 760 English Cloning of a Gane Bearing 375 3 674 English Artic Archean Subduction tered fron English Herpes Simple rus Tums CM English i Ice Binding Structure and English Amo Sportaneous Resistance to Acute English 5 Observational Evidence for English i Archean Cratoric Roots Mantle English TE Defects Axonal Transpart in a English e Allosteric Transition Imermedates English A Glycogen Syrthase Kinase 3 and English Artic RPA Invobement in the English Defecte T Cell Receptor English Direct Dernonstration of an English 4 Immunelogical Function of a English Mating Cost of Large Floral English ic WAH Gens Required for English Atie Large Size of Lyman Alpha Gas English Unexpected Paitems of Parentage English cl Friction Meh Distribution in a English artic Effect of Bed Morphology on Flow NOW Nw wo LM _ 4 O1 an amp ON O9 Y ui w nn wm no m mw amp
89. ques in the evaluation of research programs Science and public policy Vol 14 No 2 April 1987 pp 99 20 Francis Narin Evaluative bibliometrics The use of publication and citation analysis in the evaluation of scientific activity Contract NSF C 627 National Science Foundation March 31 1976 Monograph 456 NTIS Accession PB252339 AS cited in F Narin Bibliometric techniques in the evaluation of research programs Science and Public Policy Vol 14 No 2 April 1987 note 3 21 Cf L Leydesdorff The relation between qualitative theory and scientometric methods in science and technology studies Scientometrics Vol 15 No 5 6 1989 pp 333 347 22 The impact factor was developed by ISI which still ensures its dissemination through an annual bibliometric publication called Journal Citation Reports There are many ways of calculating the impact factor This is the ISI definition For a description of other ways of calculating the impact factor see E Garfield Citation analysis as a tool in journal evaluation Science 178 471 79 1972 A discussion of impact factors will be found in P O Seglen How representative is the journal impact factor Research evaluation Vol 2 December 1992 pp 143 149 The problem of standardisation is dealt with in Y Gingras Performance indicators Keeping the black box open Proceedings of the Second International Conference on the Impact of R D Ottawa 1995 20 Bibliometrics Applied to
90. ration A literature review LISR 15 1993 pp 325 354 HERBERTZ H Does it pay to cooperate A bibliometric case study in molecular biology Scientometrics Vol 33 No 1 1995 pp 117 122 HICKS D and J Sylvan KATZ Science policy for a highly collaborative science system Science and Public Policy Vol 23 No 1 February 1996 pp 39 44 KATZ J S Bibliometric assessment of intranational university university collaboration D Phil thesis Science Policy Research Unit University of Sussex 1993 KATZ J S and B R MARTIN What is research collaboration Research Policy Vol 26 1997 pp 1 18 LECLERC M and P DUFOUR International science and technology collaboration in J de la Mothe and P Dufour Eds Science and technology in Canada Harlow Essex Longman Publishing 1993 pp 125 161 LECLERC M and J GAGNE International scientific cooperation The continentalization of science Scientometrics Vol 31 No 3 1994 pp 261 292 LUUKKONEN T R J W TIJSSEN O PERSSON and G SIVERTSEN The measurement of international scientific collaboration Scientometrics Vol 28 No 1 1993 pp 15 36 LUUKKONEN T O PERSSON and G SIVERTSEN Understanding patterns of international scientific collaboration Science Technology amp Human Values Vol 17 No 1 1992 pp 101 126 MELIN G The networking university A study of a Swedish university using institutional co authorships as an indic
91. rd analysis are described in terms of the bibliometric monitoring of research The great advantages of the use of bibliometric data are availability and flexibility There are many applications it is important to remember that bibliometric studies always make use of several indicators and sometimes of more than one type of data e g when publications and patents are combined While numerous applications are in fact still recent in terms of evaluation and decision making In recent years bibliometric tools have proven their usefulness The next step is for them to be accepted as a complement to traditional evaluation methods and decision making techniques The great potential of bibliometrics applied to the study of scientific and technological research is only described in general terms here Readers who wish to know more about bibliometric methods may consult the accompanying annotated bibliography They will find there a detailed account of bibliometric tools as well as many examples of applications in several fields and many descriptions of national experiences In closing it should be noted that the bibliographic database of the Observatoire de la science et de la technologie is now available for the analysis of Canadian scientific output An analysis of aggregate statistics produced using the database is also available 37Benoit Godin Yves Gingras and Louis Davignon Knowledge flows in Canada as measured by bibliometrics Worki
92. re des Sciences et des Technologies UNIV UNIV McGILL and McGILLL UNIV the latter probably as a result of erroneous data entry Consequently standardisation brought about a reduction of the 4 524 Canadian institutions in the Table A Types of documents Canadian authors in 1995 Types sources SCI SSCI AHCI Total article 22 997 3 891 1 129 28 017 66 4 meeting abstract 3 851 812 4 4 667 book review 3 1 397 2 183 3 583 note 2 015 198 34 2 247 letter 1 148 105 23 1 276 review 870 160 55 1 085 editorial material 646 39 23 708 editorial 145 discussion correction poetry biographical item bibliography software review reprint music score review record review fiction news item art exhibit review excerpt film review hardware review database review TV radio review theatre review Total 31 78 3 60 42 16 Source Observatoire des sciences et des technologies CIRST addresses to 2 839 institutions in the database i e a decrease of 37 This procedure ensured that each institution was identified correctly under a single title Errors caused by incorrect addresses can affect the results and therefore the evaluation of national performance The quality of an evaluation depends in part on the quality of the bibliometric data with the measurement of national research performance Science and public policy Vol 15 No 3 June 1988 149 152 APPENDIX II Specification of the database of the Observatoire des
93. result is greater than 1 0 the index shows that the country in question produces more than its share of the publications in that discipline compared to the rest of the world The same type of calculations can be applied to provinces cities or sectors Example 2 shows the specialisation indices for Canada in the natural sciences engineering and the biomedical sciences 3 Flow analysis Flow indicators emphasise the relationships between researchers institutions and research fields As a result they are sometimes called relational indicators Science and technology are the result of exchanges of knowledge and collaborative work among researchers It is therefore necessary to establish indicators to identify these exchange networks Flow analysis is an attempt to describe existing relationships among the various actors of a given system Flow measurements can focus on the relationships between individuals institutions targeted sectors of activity e g university industry relationships linkages between science and technology or even collaborative efforts between provinces and countries Flow analysis provides an important measure of the integration of researchers and of the scope of a national network of collaboration both types of information help identify the effectiveness of public intervention and its impact The database of the Observatoire des Sciences et des Technologies was designed specifically to determine the flow patterns in the n
94. rnationale du Qu bec et du monde 1980 1990 Qu bec Minist re de l industrie du Commerce de la Science et de la Technologie September 1995 p 337 30 M P Carpenter and F Narin The Adequacy of the Science Citation Index SCI as an indicator of international scientific activity Journal of the American Society of Information Science Vol 32 No 6 1981 pp 430 439 31 Y Okubo Bibliometric indicators and analysis of research systems Methods and examples Paris OECD 1997 STI Working Papers 1997 1 18 Bibliometrics Applied to Public Policy Methods and Examples 27 of countries is maintained TABLE C shows that the SCI database provides good coverage of publications by Canadian and Quebec authors in comparison with the French database PASCAL which nevertheless surveys more periodicals On the other hand Canada s scientific production in the social sciences and humanities is less well represented in the SSCI and AHCI databases The research subjects in these disciplines are in some ways an obstacle to proper international coverage There is in fact throughout the world a certain consensus about the subjects methods and quality of research in the hard sciences This consensus internationalises so to speak the scientific development of the hard sciences On the other hand research in the humanities and social sciences is more contextual Research subjects depend on the location and the society in which they appear For this reas
95. s Nursing Forensic medicine Clinical neurology Rehabilitation Public health Biomedical social sciences Discipline Political sciences International relations Political science Discipline Administrative sciences Public administration Business Business Finance Management Industrial and labour relations Discipline Religious sciences Religion Discipline Sociology Social issues Sociology Discipline Social work Social work Discipline Economics Economics Discipline Education Education Special education Education and educational research How to Order Catalogued Publications These and other Statistics Canada publications may be purchased from local authorized agents and other community bookstores through the local Statistics Canada offices or by mail order to Statistics Canada Operations and Integration Division Circulation Management 120 Parkdale Avenue Ottawa Ontario KIA 0T6 1 613 951 7277 National toll free order line 1 800 267 6677 Fax number 1 613 951 1584 Toronto Credit Card only 416 973 8018 CATALOGUED PUBLICATIONS Statistical Publication 88 202 XPB Industrial Research and Development 1997 Intentions with 1996 preliminary estimates and 1995 actual expenditures 88 204 XPB Federal Scientific Activities 1997 98 annual 88 001 XPB Science Statistics monthly Volume 21 No 1 Scientific and Technological S amp T Activities of Provincial Governments 1987 88 to 1995 96 N
96. s applied to the content of articles and thus provides a more in depth study of research topics than does citation analysis Co word analysis is a method that identifies links between research topics according to the simultaneous presence of key words within documents The goal is to provide a graphical representation of the structure of scientific output at a given time in a two dimensional or three dimensional space Such a graphical representation is based on plotting maps showing the position of the content of scientific output as well as the people involved in research Discipline analysis is one of the applications of co word analysis It helps pinpoint the scientific and technological output of researchers within the more limited framework of a discipline Co word analysis and co citation analysis can serve as a research management tool to the extent that they help monitor the development of research topics within a given institution or program In the context of an evaluation they are used to measure scientific and technological output in terms of the stated research priorities of policies and programs 6 Representativeness of bibliographic sources The database of the Observatoire de la science et de la technologie provides good coverage of all the fields of knowledge However the use of ISI databases as the main source of information for bibliometric studies raises a number of issues concerning the representativeness of the data Sever
97. s of information Table A indicates the types of documents covered by each database After the ISI data were gathered the names of Canadian institutions contained in the addresses were standardised This was done because the information contained on the ISI discs included errors and significant variations Thus McGill University was recorded in three different ways McGILL 38 The data were selected on the basis of the year of publication and not on that of the year of the databases used 39 A retrospective version is available in hard copy going back to 1945 40 E Garfield How ISI selects journals for coverage Quantitative and qualitative considerations Current Contents No 22 May 28 1990 41 E Garfield How to use the Social Sciences Citation Index Current Contents Vol 27 No 12 13 July 2 1984 42 Arts amp Humanities Citation Index Journal Citation Report Philadelphia Institute for Scientific Information Inc 1995 43 A discussion of errors in the addresses will be found in R E de Bruin and H F Moed The unification of addresses in scientific publications in L Egghe and R Rousseau Eds Informatrics 89 90 Amsterdam Elsevier Science Publishers B V 1990 65 78 J Anderson et al On Line approaches to measuring national scientific output A cautionary tale Science and public policy Vol 15 No 3 1988 53 161 L Leydesdorff Problems A 4 APPENDIX II Specification of the database of the Observatoi
98. scipline Arts amp humanities Arts and humanities in general Discipline Others Demography Environment Family studies Mathematical methods in the social sciences Interdisciplinary social sciences Substance abuse Transportation Environmental studies Asiatic studies Women s studies Discipline Fine Arts Architecture Art Dance Music Theatre Discipline Library science Information science amp library science Discipline Communications Communications Discipline Law Criminology and penology Law Discipline Geography development and urban studies Geography Planning and development Regional studies Urban studies Discipline History History History of the social sciences History and philosophy of science Discipline Linguistics Linguistics Language and linguistics Discipline Literature Classics Literary Review Literature British Isles Literature Romance Literature African Canadian Australian literature American literature German Scandinavian Dutch literature Slavic literature Poetry Discipline Philosophy Philosophy Discipline Psychiatry Psychiatry Discipline Psychology Psychology Applied psychology Clinical psychology Developmental psychology Experimental psychology Mathematical psychology Social psychology Psychology biology Psychology psychoanalysis Psychology education Discipline Health Ergonomics Geriatrics and gerontology Health policy amp service
99. significance within the context of science policy The paper by Beaver and Rosen describes the history of the phenomenon These authors maintain that the development of scientific collaboration is linked to the institutionalisation of scientific research All these bibliometric studies illustrate the persistence and scope of the co author analysis The paper by Hicks amp Katz provides an interesting perspective on the need to integrate scientific collaboration within the framework of research policy BEAVER D de B and R ROSEN Studies in scientific collaboration Part 1 The professional origins of scientific co authorship Scientometrics Vol 1 No 1 1978 pp 65 84 BRAUN T I GOMEZ A MENDEZ and A SCHUBERT International co authorship patterns in physics and its subfields 1981 85 Scientometrics Vol 24 1992 pp 181 200 CARPENTER Mark P and Francis NARIN The adequacy of the Science Citation Index SCI as an indicator of international scientific activity Journal of the American Society of Information Science Vol 32 No 6 1981 pp 430 439 DALPE R International activities of public laboratories in Canada Technology in Society Vol 19 No 2 1997 pp 127 143 Bibliography 39 FRAME J D and M P CARPENTER International research collaboration Social Studies of Science 9 1979 pp 481 497 HARSANYI Martha A Multiple authors multiple problems bibliometrics and the study of scholarly collabo
100. t the international level ANDERSON F and R DALPE S amp T indicators for strategic planning and assessment of public research institutions Knowledge and Policy The international journal of knowledge transfer and utilization Vol 9 No 1 Spring 1996 pp 49 69 ANDERSON F and R DALPE The evaluation of public applied research laboratories Canadian Journal of Program Evaluation Vol 6 No 2 1991 pp 107 125 BECK M T and V GASPAR Scientometric evaluation of the scientific performance at the faculty of natural sciences Kossuth Lajos University Debrecen Hungary Scientometrics Vol 20 No 1 1991 pp 37 54 Bibliography 47 COLMAN A M D DHILLON and B COULTHARD A bibliometric evaluation of the research performance of British university politics departments Publications in leading journals Scientometrics Vol 32 No 1 1995 pp 37 48 COLLINS P Quantitative assessment of departmental research A survey of academics views London SEPSU Policy Studies No 5 1991 DALPE R and E GAUTHIER Evaluation of the industrial relevance of public research institutions Research Evaluation Vol 3 No 1 pp 43 54 DAVIS G and P ROYLE A comparison of Australian university output using journal impact factors Scientometrics Vol 35 No 1 1996 pp 45 58 HERBSTEIN F H Measuring publications output and publications impact of faculty members of a university chemistry department S
101. ted indexes Science Citation Index Social Sciences Citation Index and Arts and Humanities Citation Index to ensure that all institutional addresses were standardized and assigning a sector code university government business etc to each institution The three working papers are part of the project In order to facilitate the use of bibliometric information for policy and decision making a series of regional workshops is being held to introduce the database and to discuss uses of it TABLE OF CONTENTS A BIBLIOMETRICS APPLIED TO PUBLIC POLICY METHODS AND EXAMPLES ss Y T Bibliometrie indieal rsi neke ante lisniel niet Adi ii idad 1 1 Descriptive indicators 1 2 Relational indicators 2 Analysis of scientific output 3 Flow analysis ss 3 1 International flow patterns 3 2 National flow patterns 4 Research evaluation ecccccicdcccssccdccvncietccsticctecunactecal a daa iii 4 17 Impact meaSurements misent ere et Aaa 5 Bibliometric monitoring of research ss 5 1 Administrative bibliometric monitoring sise 6 Representativeness of bibliographic sources ss 1 General sources on bibliometrics 2 Bibliometrie statisties rte Na tase tad Caco MN eae ae ee aad Bs INdicalors seele nn ne nn E 3 1 Indicators in general 3 2 Indicators of scientific collaboration 4 Evaluation oossoo 4 1 Research evaluation 4 2 Program evaluation 4 3 Evaluating nati
102. tion of basic research programs projects IEEE Transactions on Engineering Management Vol 30 No 3 1983 pp 106 112 GIBBONS M and L GEORGHIOU Evaluation de la recherche un choix de pratiques en vigueur Paris OCDE 1987 GEORGHIOU Luke Evaluation of research Synthesis report Ad hoc Group on Scientific and University Research Directorate for Science Technology and Industry Paris OECD January 1986 HELANDER E Evaluation activities in the Nordic countries Scientometrics Vol 34 No 3 1995 pp 391 400 42 Bibliography KOSTOFF R N Federal research impact assessment Axioms approaches applications Scientometrics Vol 34 No 2 1995 pp 163 206 KOSTOFF R N Performance measures for government sponsored research Overview and background Scientometrics Vol 36 No 3 1996 pp 281 292 LINK A N Economic performance measures for evaluating government sponsored research Scientometrics Vol 36 No 3 1996 p 325 LUUKKONEN T Bibliometrics and evaluation of research performance Annals of Medecine Vol 22 No 3 1990 MARTIN B R The use of multiple indicators in the assessment of basic research Scientometrics Vol 36 No 3 July 1996 pp 343 362 MARTIN B R and J IRVINE Assessing basic research Some partial indicators of scientific progress in radio astronomy Research Policy Vol 12 No 2 1983 pp 61 90 MOED H F et al A comparative study of bibliometri
103. tly with the activities of universities and research centres 5 1 Administrative bibliometric monitoring The problems which administrators of research institutions have to overcome are different from those of public policy makers Administrators need to access detailed information about the research activities for which they are responsible Bibliometric monitoring helps them understand and evaluate research activities taking place within research units Administrators use bibliometrics for guidance about the future of research on the basis of existing research in a given field Strategic positioning is one of the major applications of bibliometric indicators This is done by identifying through the use of databases the people who are active in a field the topics on which they are working and the networks within which they operate This information is used to identify emerging niches those that have reached maturity as well as those in decline along with the actors and partners who are responsible for research output Research managers use data on the relationships between researchers and the links between research topics to understand their position within the network and to evaluate it in terms of what is going on in a given field of research 24 Bibliometrics Applied to Public Policy Methods and Examples Example 9 State of nanotechnological research in Canada In 1996 the National Research Council of Canada commissioned a biblio
104. tor Dr F D Gault 613 951 2198 An Information System for Science and Technology Chief Indicators Development Dr Frances Anderson 613 951 6307 Chief Research and Analysis Michael Bordt 613 951 8585 Chief Data Integration Projects Daood Hamdani 613 951 3490 Project Development Officer Antoine Rose 613 951 9919 Science and Technology Section Acting Project Leader Private Sector Don O Grady 613 951 9923 Project Leader Public Sector Bert Plaus 613 951 6347 Senior Project Officer Janet Thompson 613 951 2580 FAX 613 951 9920 Working Papers The Working Papers publish research related to science and technology issues All papers are subject to internal review The views expressed in the articles are those of the authors and do not necessarily reflect the views of Statistics Canada PREFACE This paper Bibliometric Analysis of Scientific and Technological Research A User s Guide to the Methodology by Elaine Gauthier provides an overview current usage of bibliometric methods and techniques including an extensive bibliography It also provides technical specifications on the database of Canadian authors that has been developed with Statistics Canada support by the Observatoire des Sciences et des Technologies This working paper is a companion document to two other working papers The first Knowledge Flows in Canada as Measured by Bibliometrics uses the database to develop statistical indicators of k
105. tors Leiden The Netherlands 14 16 November 1988 Leiden DSWO Press Science Studies Series No 3 1989 38 Bibliography Van RAAN A F J and R J W TIJSSEN An overview of quantitative science and technology indicators based on bibliometric methods TEP Conference Paris OECD July 1990 STEAD H Collection of S amp T statistics Science and Public Policy Vol 19 No 5 pp 275 280 ZITT M and N TEIXEIRA Science macro indicators Some aspects of OST experience Scientometrics Vol 35 No 2 1996 pp 209 222 3 2 Indicators of scientific collaboration The documents listed in this section deal mainly with the empirical study of scientific collaboration There are contributions on international collaboration as well as on networks bringing together researchers and institutions They all share a common element the use of co author analysis This provides a good overview of the range of possibilities Generally speaking international collaboration is a major topic The papers by Melin and by Peters and van Raan deal more specifically with the links between academic institutions There is also an original contribution by Qiu who uses co author analysis to study links between disciplines Subramanyam provides a broader outlook by reviewing the many interpretations of the concept of scientific collaboration in terms of the sociology of science whereas Katz and Martin discuss the concept of scientific collaboration and its
106. tors of science and foreign dependence World Patent Information Vol 5 No 3 1983 pp 180 185 CARPENTER M P and F NARIN Assessment of the linkages between patents and fundamental research Paris OECD Patents and Innovation Statistics Seminar June 28 30 1982 COWARD H R and J J FRANKLIN The interaction of science and technology Exploring the relationship of a bibliometric model to patents Center for Research Planning 1985 NARIN F Patents bibliometrics Scientometrics Vol 30 No 1 1994 pp 147 155 NARIN F Patents indicators for the evaluation of industrial research output Scientometrics Vol 34 No 3 1995 pp 489 496 NARIN F M P CARPENTER and P WOOLF Technological performance assessments based on patents and patent citations IEEE Transactions on Engineering Management EM 31 No 4 1984 pp 172 183 NARIN F E NOMA and R PERRY Patents as indicators of corporate technological strength Research Policy Vol 16 1987 pp 143 155 NAUWELAERS C and A REIDS Methodologies for the evaluation of regional innovation potential Scientometrics Vol 34 No 3 1995 pp 497 511 OECD Using patent data as science and technology indicators Patent Manual 1994 Paris OECD 1994 Bibliography 57 PAVITT K Patent statistics as indicators of innovative activities Possibilities and problems Scientometrics Vol 7 No 1 2 1985 pp 77 99 RABEHARISOA V A SIGOGNEAU and F
107. vincial Distribution of Federal Expenditures and Personnel on Science and Technology 1987 88 to 1995 96 October 1997 Price 75 00 Commercialization of Intellectual property in the Higher Education Sector A Feasibility Study October 1997 Price 75 00 Business Demographics as Indicators of Innovation Activity October 1997 Price 75 00 Methodology for Estimation of Higher Education R amp D Personnel November 1997 Price 75 00 ST 97 14 Estimates of Research and Development Personnel in Canada 1979 1995 November 1997 Price 75 00 WORKING PAPERS 1998 ST 98 01 ST 98 02 ST 98 03 ST 98 04 ST 98 05 ST 98 06 A Compendium of Science and Technology Statistics February 1998 Exports and Related Employment in Canadian Industries February 1998 Job Creation Job Destruction and Job Reallocation in the Canadian Economy February 1998 A Dynamic Analysis of the Flows of Canadian Science and Technology Graduates into the Labour Market February 1998 Biotechnology Use by Canadian Industry 1996 March 1998 An Overview of Statistical Indicators of Regional Innovation in Canada A Provincial Comparison March 1998 RESEARCH PAPERS 1996 AND 1997 No 1 No 2 No 3 No 4 No 5 The State of Science and Technology Indicators in the OECD Countries by Benoit Godin August 1996 Knowledge as a Capacity for Action by Nico Stehr June 1996 Linking Outcomes for Workers to Changes in Workplace Practices
108. ws in Canada as measured by bibliometrics Working paper Science and Technology Redesign Project Statistics Canada 1998 2 OECD Proposed standard practice for surveys of research and experimental development Frascati Manual Paris 1994 3 Leclerc Michel et Gingras Y Les indicateurs du financement priv de la R amp D universitaire au Qu bec critique de la m thode Revue canadienne de l enseignement sup rieur vol 23 n 1 1993 pp 74 107 4 Government of Canada Science and technology for the new century A federal strategy Ottawa 1996 5 OECD and Eurostat Proposed guidelines for collecting and interpreting technological innovation data Oslo manual Paris 1997 8 Introduction Productivity aimed at operationalizing the concept of national innovation system NIS A national innovation system is defined as a set of interrelated actors whose activities are oriented towards the development of new products processes and services Nowadays governments are turning their attention to the dynamics of national innovation systems notably the flow and transfer patterns between various actors Indicators linked specifically to this new aspect of science and technology i e flow patterns are practically non existent The challenge associated with this new concept is to operationalize the notion of national innovation system Bibliometrics is a tool that can be used for this purpose But before we deal with flow measurement

BIBLIOMETRIC ANALYSIS OF SCIENTIFIC AND TECHNOLOGICAL

Contents

Download Pdf Manuals

Related Search

Related Contents