Home

CogSketch Manual - Qualitative reasoning group

1. Tutorials Cogsketch General Tutorial User Manual Design Coach Advanced Tutorial For About CogSketch Fl Design Coach Basic Tutorial Subsketches And The Metalayer Open Worksheet Worksheet Basic Tutorial OpenaCogSketch Worksheet file lt 7 3 3 COGSKETCH ESSENTIALS Start CogSketch using the shortcut made in your Start Programs menu CogSketch will open its knowledge base and provide you with an empty workspace From the File menu you have five options to open or start a sketch Open Sketch New Sketch New Worksheet New Design Sketch and New Perceptual Sketch Open Sketch opens sketch files saved to disk New Worksheet allows you to create a new CogSketch worksheet while New Sketch makes a new basic sketch Worksheets are useful for researchers wanting to design controlled experiment sketches and teachers designing problems for students along with solutions that can be used with the CogSketch tutor New Perceptual Sketch opens the experimental perceptual sketchpad described in Section ale In this example we ll be making a basic sketch so choose New Sketch from the File menu This will create a new empty sketch Sketch 1 Subsketch 1 mE n 4 ink vt ye L Select a glyph or ink to edit its properties Current Subsketen Subsketch 1 CogSketch version 3 07 11 Now that CogSketch is up and running you can start to draw Each eleme
2. D 0 ge Available Concepts Stort typing in the box above to search the available collections Name Object 7699 Ink Properties calar style gt Properties Glyph Type entity Assigned Concepts House Modern X House Modern House Country Houseboat HouseFly HouseFuse ahal 13 4 Farther down the properties sidebar 15 a field labelled Name Use this to give your 5 glyph a more readable name This name will also appear on the sketch under your glyph house tree bird Subsketch 1 Ben finish eG 7 1 5 Ph M Propertie Glyph Type entity Assigned Concepts House Modern N Available Concepts Start typing in the box above to search the available collectians Name My housel Ink Properties calar style gt ill The Ink Properties section below that lets you change the color line style and line thickness of any selected ink Here are some tips to help you use these facilities more effectively The activities one does when drawing glyphs versus specifying their content feel quite different Consequently many people find it best to draw for a while steps 1 and 2 then specify the collections for all of the glyphs they have created step 3 cycling back and forth between these two activities as desired CogSketch does not automatically decompose glyphs into smaller pieces bu
3. CogSketch version 3 07 28 Glyph Properties relations Worksheets and Design Coach Sketches e AccelerationArrow e AppliedForceArrow e AssumedForceArrow e DirectionOfMovement e DirectionAnnotation e ForceAtrow e FrictionalForceArrow CogSketch version 3 07 GravitationalForceArrow InitialVelocityArrow NetForceArrow NormalForceArrow RotationalForceArrow TensileForceArrow VelocityArrow 29 4 4 EDITING ITEMS ON THE SKETCH 4 4 1 SELECTING ITEMS The Glyph Select tool lets you select and edit glyphs You select items either by drawing a lasso around them ot by clicking on them The picture below shows a lasso selection being drawn You can only select glyphs on the current layer Layer 1 in the picture below You can select as many glyphs on the current layer as you want Tapping or lassoing a selected glyph unselects it so you can add to or subtract from your selection as desired In other words each time you click or lasso a glyph it toggles whether or not it s selected EI tree sun Subsketch 1 co fame S g O Q x Pd eine uai n v X color style QU x 3 Properes 0 00 0 new new subsketch layer Select a glyph or ink to edit its properties 1 leaves Current Subsketch Subsketch 1 0 1 1 3 in zoom 10056 The Ink Select tool is used to select the ink within the glyphs You use this tool when you want to edit indiv
4. RA layers Wright Wright Dontae 4 7 2011 82 0 100 There are a couple ways to add sketches you can either use drag and drop dragging sketches from the Windows file explorer or you can use the Add sketches button and select the sketches you want to add from a file browsing dialog Either way you will be prompted for further information about each sketch added If you re viewing the sketches for a given assignment when you add a sketch the Assignment field will already be set to that assignment for you If you re viewing the sketches for a student the Students field will already have that student selected for you CogSketch version 3 07 60 The Date received field is automatically set to the file date of the sketch file that you re adding You can of course change the values of any of the automatically populated fields if you want The Students field shows a list of all the students defined for the class and provides check boxes so that you can select which student or students submitted the sketch To explain the Solution Sketch and Score fields let s talk a bit about grading Through the gradebook you can use the CogSketch tutor plus the grading criteria defined for a worksheet via the worksheet property editor described in the section titled Designing a Worksheet to automatically grade worksheet sketches submitted by students If you wish to do this one sketch in each assignment is treated as the
5. and a smaller circle at the end of the stroke as shown below 4 7 7 3 Sketch Playback Allows researchers to playback the sketch as it was drawn Currently the playback only shows ink being drawn it does not show other user actions like moving resizing or deleting CogSketch version 3 07 45 4 7 7 4 Store Current Sketch As Case Stores the current sketch as a case in a case library The case library is specified by the user You can also specify that the contents should be dumped as a flat file which can subsequently be loaded into other FIRE based reasoning systems 4 7 7 5 Export Sketch Knowledge Dumps all the knowledge about the sketch to a text file The format used is MELD style case sensitive predicate calculus expressions The relation ist Information is used to contextualize knowledge about a sketch Each subsketch is its own microtheoty with the microtheory for the entire sketch being in their logical environments 4 7 7 6 Export Sketch as Image Saves a bitmap file to disk with only the currently visible layers shown 4 7 7 Export Ink Saves a data file describing the ink strokes including timing information to a comma separated value file The first line of the file contains headings for the columns subsequent lines each describe one piece of ink See Appendix C Exported Ink File Format for details about the file format 4 7 7 8 Export Sketch History Creates a data file describing the history of
6. gives the human readable name for the sketch The complete list of data columns in the file is as follows Sketch Namestring human readable name of the sketch Sketch Case name of the microtheory case associated with the sketch Subsketch Namestring human readable name of the Subsketch Case name of the microtheory case associated with the subsketch Layer Namestring human readable name of the Layer Objname internal name of the layer Glyph Namestring human readable name of the CogSketch version 3 07 88 e Glyph Objname internal name of the object represented by the glyph e Ink ID integer ID of the ink stroke e Ink Type Describes what kind of primitive ink object is being described This is usually sketch point meaning that an individual point is being described e Ink Point X X coordinate of the point e Ink Point Y Y coordinate of the point e Ink Point Timestamp seconds The timestamps are given in seconds with millisecond precision that have elapsed since the sketch was first started When sketches are saved the current timestamp is saved along with the file so that the time progression can start off from there when the sketch is opened again For example if you create a new sketch and then start drawing exactly two minutes later the timestamps for the points in that bit of ink will start at 120 000 seconds Let s say you then save the sketch at 500s If you later re open the sketch and start drawi
7. means that the event EVENT occurs entirely within the time interval TIMEINTERVAL Thus occursDuring Seer eee eis a DNA replication occurs during cell division Name occursDuring Ink Properties a color style 4 px 25 4 3 2 5 Glyph Properties annotations Annotations can be used to specify additional information about the glyphs on your sketch To illustrate this further we ll use the following example Let s say that we want to denote the temperature of the tree s leaves in the sun Draw the glyph that you ll use to represent the temperature CogSketch version 3 07 26 Select the glyph to edit its properties Change the Glyph Type to annotation In the What is this section select TemperatureIndicator CogSketch now knows this glyph will denote the temperature of some other glyph In this example we set the value to 92 degrees Fahrenheit What is this Temperaturelndicator Value 92 degree Fahrenhei This is the collection af sketch annotations used to indicate the temperature of an object Name leaves Temperaturelndicator Temperatureindicator applies to trunk Ink Properties 4 t ground colo r styl 2 Now we have to tell CogSketch what glyph has a temperature of 92 F As soon as you changed the glyph type to annotation a blue handle was added to your glyph Drag this to the glyph
8. of a double helix DNAMolecules can be usefully represented as m Klaine eo Note that any new collections you define here are not permanently added to the knowledge base They will be specific to this worksheet and will be removed from the knowledge base when the worksheet is closed This reduces the possibility of naming conflicts across wotksheets CogSketch version 3 07 53 The next section Allowed Relations is similar It allows you to specity which relations the worksheet user can assign You can search for relations in the knowledge base or define new ones that are specific to the current worksheets The last section on this tab Allowed Annotations allows you to specity which annotations will be available While setting up the workspace subsketch you can add glyphs background images layers adjust their properties etc just like you would in any sketch These will be saved along with the wotksheet The next tab of the Worksheet Properties Editor is used when defining a solution for the worksheet This is only necessary if you intend to use the CogSketch Worksheet Tutor The solution if provided will be a subsketch Worksheets when first created do not have a solution subsketch so the tab initially looks as follows Include solution zubsketch EL il If you check the Include solution subsketch checkbox a new subsketch will be added to the sketch the solutio
9. solution sketch This is a sketch for which you the instructor has created a solution subsketch and defined grading criteria using the wotksheet property editor When adding or editing such a sketch in the eradebook the Solution Sketch should be checked This tells the gradebook that it should use that sketch as the solution when grading the students worksheets The Score fields are present when adding or editing a sketch in case you want to assign your own grade They will be filled in for you if and when you run the automatic grader Once you click the OK button a copy of the sketch file will be made and added to the gradebook By default these are added to your Windows Documents folder under cogSketch gradebook organized by class and assignment You can change the location of these sketches if you wish in the Preferences You can always edit the gradebook s information about a sketch later by selecting a sketch and clicking the Edit button or by right clicking a sketch and choosing Edit lt sketch name gt from the shortcut menu Double clicking a sketch or choosing Open lt sketch name gt in CogSketch from the right click menu will open the sketch so you can view or edit the sketch itself 7 3 GRADING WORKSHEETS CogSketch can be used to grade student worksheets automatically To do so the following conditions must be met solution sketch has been entered for the assignment e hat solution sketch contains a s
10. Mechanics In Proceedings of the 22nd International Workshop on Qualitative Reasoning Boulder CO Wetzel J and Forbus K 2009 Automated Critique of Sketched Mechanisms In Proceedings of the 21st Innovative Applications of Artificial Intelligence Conference Pasadena California Wetzel J and Forbus 2010 Design Buddy Providing Feedback for Sketched Multi Modal Causal Explanations In Proceedings of the 24th International Workshop on Qualitative Reasoning Portland Oregon Wetzel J and Forbus K 2012 Teleological Representations for Multi Modal Design Explanations Proceedings of the 26th International Workshop on Qualitative Reasoning Los Angeles California Yin P Forbus K Usher J Sageman B amp Jee B 2010 Sketch Worksheets A Sketch based Educational Software System Proceedings of the 22nd Annual Conference on Innovative Applications of Artificial Intelligence Yin P Chang M D and Forbus K D 2010 Sketch based Spatial Reasoning in Geologic Interpretation In Proceedings of the 24th International Workshop on Qualitative Reasoning Portland Oregon CogSketch version 3 07 83 13 APPENDIX A KEYBOARD SHORTCUTS Drawing e F2 Draw mode e F3 Glyph select e F4 Ink select e F5 Erase F7 Finish glyph e Ctrl Decrease line thickness e Ctrl Increase line thickness File operations e Ctrl N New sketch e Ctrl O Open sketch e Ctrl S Save e Ctrl
11. and thus will help us develop these capabilities 11 2 AUTHORING ENVIRONMENT FOR SPATIAL ROUTINES The cognitive models of visual processing in CogSketch are built in the Spatial Routines language developed in Andrew Lovett s Ph D thesis 23 We plan on implementing an integrated development environment IDE for spatial routines so that cognitive scientists can more easily write their own spatial models on top of CogSketch at a level of abstraction that should be more suitable for formulating computational models 11 3 EVERYDAY PHYSICAL SEMANTICS To support STEM education CogSketch has to have a reasonable level of understanding about how objects interact including interactions between shape forces motion and materials is necessary The relevant level of understanding is mostly qualitative detailed numerical methods and traditional simulations are not appropriate because such information is not yet specified during conceptual design You can try out the current facilities by using Design Coach a CogSketch application built into the software We expect that similar properties will hold for sketching in support of geoscience education and other STEM relevant domains We are integrating prior QRG research results e g 15 to provide this level of understanding for the contents of sketches You can access some experimental facilities from the QM menu which only appears in Developer mode currently 11 4 SPECIALIZED INTERFACES am
12. but CogSketch doesn t know what each of them represents yet j house tree bird Subsketch 1 finish S ui E glyphs ink glyph 34 XN gt new new subsketch layer i P CogSketch version 3 07 12 3 Let s tell CogSketch what each glyph means Select one of the glyphs with the glyph lasso With this tool you can either draw a lasso around the things you want to select or tap them When you have a single glyph selected its properties are displayed in the sidebar to the right of the drawing area zy house tree bird Subsketch 1 2 finish 8 E glyphs ink A pam T ew LH new subsketch layer glyph erase ink F5 Current Subsketch Subsketch 1 1 30 2 09 in zoom 11695 Worksheets and Design Coach sketches are designed with a small set of available concepts Since this 1s just a regular sketch you can choose any concept in the entire knowledge base This makes selecting the concepts a bit more complicated though since there are so many In the box labelled Available Concepts start typing house As you type CogSketch will search the knowledge base for concepts starting with the letters you typed Select House Modern and click the upwards facing arrow icon to add this concept to the Assigned Concepts CogSketch now knows your glyph represents a modern house CogSketch version 3 07 2 X
13. can compute analogical relationships between layers or subsketches Analogies are done using both the visual information in the sketch and the conceptual information entered through the glyph properties dialogue and the visual conceptual relationships 5 2 1 ANALOGY MENU 5 2 11 Compare Two Layers Subsketches These features allow you to compare two layers or subsketches on a sketch using the Structure Mapping Engine SME When you select one of the analogy options you will be asked to select the items to compare and what factors you would like to figure into the analogy A Compare Bundles minit taii ia Analogy Features Compare virus Invades Choose a Bundle Roundness Iw Sizes Layer Orientation Sizes Conn Grp To i Visual Grouping Sizes Cntd arp W Positional Relations Articulation Pts virus Invades Cell Choose a Bundle DELIA The analogy results will then be presented The table in the middle contains the correspondences found by SME i e what goes with what between the base and the target Selecting a correspondence causes the appropriate elements in the base and target displays to be highlighted The Show Raw Form check box toggles whether or not the pretty forms of a name or its internal form is displayed The Browse SME button enables one to drill down into the details of a match inspect candidate inferences and so on using a dynamically generated hyp
14. drawing When you click this any ink strokes drawn since the last time you clicked this button will be collected together and made into a new glyph 4 1 2 2 2 Undo You can use this to undo many actions performed while drawing and editing a sketch 4 1 2 3 e Redo Use this to redo actions that you ve recently undone 4 1 2 4 X Delete Selected Items When items are selected on the sketch pressing this button will remove them sz 4 1 2 5 Color Selector Use this to choose the color that will be used for ink you draw 4 1 2 6 Line Style This selects the line style for the ink you draw There are several varieties of dashed and dotted styles available in addition to solid lines D 4 1 2 7 Line Width Use this to control the thickness of the lines that you draw The units are pixels when the sketch is at its normal zoom level magnification of 1 0 4128 7 Visual Conceptual Relation Questions This triggers the visual conceptual relations questions described in Section 5 3 When you click this button a browser window will open where you can provide CogSketch with more information about how to interpret the relationships in your sketch 4 1 2 9 Spatial Processor Status Indicators Beneath the general status indicator is an area that is used to display the status of spatial processors The left eye represents the ink processor and the right eye represents the voronoi processor The numbers on either side
15. knowledge and less computational power than people do So CogSketch also needs more help from you than you might need to give a person you ate sketching with Sketching with CogSketch is not as fluent as sketching by yourself or with another person but we have found ways of making it flexible enough that people find it useful CogSketch is very much a work in progress All of its components are evolving to achieve the ambitious goals outlined above This requires sustained collaboration between the CogSketch research team and other AI researchers psychologists learning scientists and educators CogSketch is one of the products of SILC the Spatial Intelligence and Learning Center SILC provides exactly the environment for sustained collaboration needed to develop CogSketch This version has already benefited from many rounds of feedback from researchers both inside and outside of SILC We hope that you too will help us grow the software in ways that will ultimately help everyone The next two sections provide a summary of the ways CogSketch can be used and a summary of our approach Section 0 describes how to install it and Section 3 provides a quick start guide to using the software Section 4 provides a detailed description of CogSketch s interface Section 5 describes some of CogSketch s more advanced features including on demand spatial reasoning operations analogical matching expressing visual conceptual relationships and sk
16. pieces The document CogSketch_Spatial_Reasoning pdf that was included with your CogSketch installation describes our spatial reasoning in more detail B Cogsketch File Edit View Glyphs Spatial Analogy FIRE 5 1 1 SPATIAL MENU 5 1 1 1 Compute Positional Relations for This Laye Sketch Compute Positional Relations for this Layer u Compute Positional Relations for this Sketch Computes the positional relations among the glyphs in the specified layer sketch These are Compute Relative sizes computed only on demand After this option Identify Articulation Points is selected the results will appear in the knowledge known about the sketch Recompute All Spatial Relations Add Lattice to Layer 5 1 1 2 Compute Relative Sizes Computes the relative sizes of the glyphs in the sketch Sizes are qualitative and from the set tiny small medium big huge 5 1 1 3 Identify Articulation Points Identifies the articulation points in the sketch An articulation point is a point which if removed would divide up a graph into disconnected components 5 1 1 4 Recompute All Spatial Relations Forces all automatically computed spatial relations to be recomputed These include the RCC8 relations CogSketch version 3 07 48 5 2 USING ANALOGICAL MATCHING CogSketch uses the Structural Mapping Engine SME an analogical matcher 3 4 To access the analogical reasoning abilities of CogSketch use the Analogy menu You
17. related to Image A from the geometric analogy problem CogSketch version 3 07 67 A is to B as C is to Try clicking on the triangle next to Find Differences gt Diff A B Copy Paste Inspect Spatial Routine Reset Spatial Routine Do Routine Geometric Analogy Perception Routine Inspector y UM IEEE p Encode Image A Experimenter Windows Help Find Differences gt Diffs A B 0 53 1 Compare Images 0 53 m wW Finish Glyph Recognize Shape retrieved b Inferimage Image D Find differences between Images and Browse Facts The operation you ve selected is the one that compared images A and B and found the differences between them As you can see CogSketch has color coded the glyphs to identity the corresponding elements in this comparison If you d like to know more about this comparison you can click Browse Facts to open up a web browser with the list of facts describing the differences between the images or you can click Browse Comparison to browse the comparison computed by the Structure Mapping Engine You can also inspect the sub operations of this operation try clicking on the triangle next to Compare Images Note that for each pair of corresponding glyphs a Compare Shapes operation was computed to identify the relations between the shapes If you select one of these it will show you t
18. right clicking the glyph This opens a browser window and supports drill down through the justifications explaining why particular facts are believed tree sun Case 3602695312 Subsketch Subsketch 1 3602695312 Layer Positional Objectl 4 Layer Voronoi ObjectL 3 Layer Layer 2 ObjectL 5 Layer Layer 1 ObjectL 1 trunk Clarify Glyph Relationships Query WM Fact Edit KB Fact Edit Analogy Refresh Object List Browse all WM E Browse KB IE O Show ists Group By predicate none Sort By alpha Apply Grouping Sorting nuSketch Reasoner 1 GlyphFn Object 4 User Drawn Sketch Layer 1 human readable namestring leaves glyph represents Object 4 connectedGlyphGroupMember 3 facts connectedGlyphGroupTangentialConnection 2 facts entityTypesLastModifiedTime 1 fact forceLastModifiedTime 1 fact lglyphAnnotatedBy 1 fact 7 glyphAnnotatedBy GlyphFn bject 4 User Drawn Sketch Layer 1 AnnotatianGlyphFn Object 6 User Drawn Sketch Layer 1 glyphGraphCWA 2 facts glyphGraphEdgesFor 2 facts glyphRepresentsObject 1 fact hasRCC8Relation 8 facts inkLastModifiedTime 1 fact isa 5 facts CogSketch version 3 07 35 4 6 ADDING NEW LAYERS AND SUBSKETCHES TO A SKETCH Layers and subsketches on a sketch can be used to group conceptual information It is useful to think of layers as transparent sheets stacked on top of each other Layers within a subsketch can a
19. subsketch Allowed collections for glyphs or free text extry 1f relevant to the experiment Allowed relations for glyphs or free text extry if relevant to the experiment Allowed annotations for glyphs if relevant to the experiment Allotted time for each subsketch Default ink line width Initial view boundaries For some experiments just fitting the view to the current subsketch boundaries is not sufficient Whether or not the right click menu should be disabled Whether or not the user can assign their own namestrings to glyphs Initial interaction mode usually this is the ink drawing tool Audio file to be played when the subsketch 1s first shown CogSketch version 3 07 73 10 TROUBLESHOOTING CogSketch is still under development Consequently it is quite possible that some errors will be encountered The email address for bugs is cogsketch bugs cs northwestern edu When submitting a bug report please include the following 1 A description of what went wrong in as much detail as you can mustet 2 summary of information about machine you were using 1 operating system amount of RAM and free disk space CPU speed 3 The sketch you were working on 4 The error log Where the error log resides depends on which operating system you are using Vista Windows 7 Window 8 ProgramData QRG CogSketch error logs XP 2000 NT Documents and Settings All Users Application Data QRG CogS
20. subsketch A more complex sketch might have several subsketches For example a sketch of a complicated machine or building might include one subsketch describing its abstract structure other subsketches focusing on specific aspects of the structure with yet other subsketches describing functional subsystems site placement etc A sketch depicting a process unfolding over time or a storyboard might use a subsketch to describe each state of the process or narrative In human to human interactions the relationships between subsketches is often mentioned verbally e g the electrical system marked by glyphs e g arrows denoting transitions between steps of a process or understood via shared conventions e g cutaway drawings In CogSketch we use glyphs to introduce relationships between subsketches in the metalayer described above 4 3 ADDING ENTITIES TO A SKETCH To add an entity to your sketch you must create a glyph to represent it Simply drawing the object on the screen does not provide enough information to CogSketch You also need to tell CogSketch what parts of your drawing go together to represent one object and what object they represent The first problem is called segmentation and the second 15 called conceptual labeling CogSketch version 3 07 20 4 3 1 DRAWING GLYPHS The drawing tool is used to draw ink ink Ink strokes that you draw are accumulated as free ink until you hit the Finish Glyph butt
21. that sketch and for each layer on that sketch with layers inheriting from subsketches which in turn inherit from the sketch This partitioning of the facts inside it is what keeps the student and teacher sketches distinct in worksheets and enables sequences and processes to be represented via multiple states in the metalayer nuSketch The sketch understanding architecture underlying CogSketch Polyline A list of points which can be considered as a list of line segments This is the geometric part of digital ink Digital ink also has color and width sKEA The sketching Knowledge Entry Associate the precursor to CogSketch sKEA was the first open domain sketch understanding system CogSketch version 3 07 86 Subsketch A portion of a sketch that is treated as a unit Intuitively they correspond to separate drawing areas within a larger sketch Sometimes they are called bundles because they are groups bundles of layers each of which are spatially registered with each other Voronoi processor A software process which calculates Voronoi diagrams for layers Voronoi diagrams are used to estimate proximity in some CogSketch operations CogSketch version 3 07 87 15 APPENDIX C EXPORTED INK FiLE FORMAT The files saved by the Export Ink option on the File menu are comma separated value files they contain rows of data where each row tepresents a single ink point and the data within the rows are separated by commas T
22. the Cognitive Science Society Forbus K Ferguson R amp Usher J 2000 Towards a computational model of sketching Proceedings of OR2000 Morelia Mexico Forbus K Hinrichs T de Kleer J amp Usher J 2010 FIRE Infrastructure for Experience based Systems with Common Sense AAAI Fall Symposium on Commonsense Knowledge Arlington VA Forbus K Lockwood K Klenk M Tomai E amp Usher J 2004 Open domain sketch understanding The nuSketch approach AAAI Fall Symposium on Making Pen based Interaction Intelligent and Natural Washington DC Forbus K Tomai E amp Usher J 2003 Qualitative spatial reasoning for visual grouping in sketches Proceedings of the Seventeenth International Workshop on Qualitative Reasoning Brasilia Brazil Forbus K amp Usher J 2002 Sketching for knowledge capture A progress report IUI 02 San Francisco California Forbus K Usher J amp Tomai E 2005 Analogical learning of visual conceptual relationships in sketches Proceedings of AAAI 05 Forbus K Usher J amp Chapman V 2003 Qualitative spatial reasoning about sketch maps Proceedings of the Fifteenth Annual Conference on Innovative Applications of Artificial Intelligence Acapulco Mexico Forbus K Gentner D amp Law K 1995 MAC FAC A model of Similarity based Retrieval Cognitive Science 19 2 pp 141 205 CogSketch version 3 07 81 13 Forbus K Ushe
23. the order they were created Each subsketch can have different instuctions To try out your experiment just save the experiment sketch and open it again You will be asked if you want to run the experiment 9 2 DESIGNING MORE COMPLEX EXPERIMENTS As mentioned before we don t yet have an authoring environment for creating experiment sketches that can make use of all the capabilities that CogSketch has available Email us at cogsketch feedback cs northwestern edu and we can help you create such experiments At present the following can be configured e For the sketch as a whole o Overall experiment instructions These are displayed on the opening screen where the participant s ID is entered CogSketch version 3 07 72 O Instructions that should be shown to the participant at the conclusion of the experiment Ordering of the subsketches within the experiment Each subsketch will be presented to the subject in order So each subsketch 15 basically one task within the experiment Subject name prompt Some experimenters wanted custom text for this prompt e For each subsketch O We have a variety of user interface skins available that control whether or not various capabilities of CogSketch are available when running the experiment The sample experiment sketch described above has a very simple skin Participants can only draw undo redo delete and create glyphs Instructions shown to the participant for the
24. this site Subsketches used to be called bundles but that term is deprecated CogSketch version 3 07 7 cloud ww Se gt evaporation Sheridan Rd Vcc Noyes St When drawing a map one is thinking in terms of a top down view with a typical coordinate system expressed in terms of north south east and west Genre 15 used to tell CogSketch the overall type of what is being drawn Something geospatial something physical or something abstract When the genre is abstract no inference is made about the spatial relationships of the contents based on the spatial relationships of the ink Otherwise genre sets what spatial relations CogSketch uses with the contents of glyphs up down left right for physical descriptions north south east west and up down for geospatial descriptions When drawing a physical system one is sometimes taking a top down view but often taking a sideways view with up in the sketch corresponding to up in the world and with gravity acting downward Pose tells CogSketch how to interpret sketch coordinates in terms of the kind of thing being drawn The three options are from the top from the side and abstract The default genre and pose for new layers is Physical View Looking from the side Both genre and pose can be set by using the Layer Properties menu which can be reached via right click on the layer s tab You can step back and take a high level view of all of your subsketches at on
25. to your system and a shortcut to CogSketch will be added to your Start Programs menu Please note Depending upon the security settings on your computer you may need to be logged in as Administrator on your local domain If you have trouble installing please ensure that you are logged in with this level of access as the first step in debugging It is a good idea to back up the knowledge base The FIRE menu has operations for backing up and restoring the knowledge base 2 3 UNINSTALLING COGSKETCH If you wish to remove CogSketch from your system open the Window s control panel from the Settings entry on your Start menu and pick Add Remove Programs Find CogSketch in the list of installed programs and press the Remove button The CogSketch installer will guide you through the uninstall process FIRE the reasoning engine that CogSketch uses 6 CogSketch version 3 07 9 3 GETTING STARTED WITH COGSKETCH This section provides an introduction to some of the basic concepts of CogSketch via a simple walk through Importantly CogSketch has interactive tutorials that many will find more fun than reading this manual Others prefer manuals though and it can be useful to have a convenient place to look for answers to specific questions A note about screen shots CogSketch is being developed for multiple purposes so the interface has skins which expose different capabilities according to how it is intended to be used The men
26. user actions taken to create the sketch This includes actions like the drawing of ink strokes moving glyphs resizing glyphs changing knowledge about a glyph deleting items adding layers etc Timing information is recorded for each action with millisecond resolution Please see Section 15 for a description of the format used for this file 4 7 7 9 Create History Report Creates an HTML document detailing the history of user actions while working with the sketch You can select which actions to include and select for which actions a screenshot of the sketch should be included in the report for example you might want to see what the sketch looked like each time the user requested tutor feedback for a worksheet This can be vety useful for those running experiments It creates a single easy to read document with pictures showing what the subject s sketch looked like at various key points during the experiment 4 7 7 10 Batch Image Export Performs the Export Sketch as Image operation on each sketch in a specified folder directory For more information about microtheories please see http www cyc com cycdoc course contexts basic module html CogSketch version 3 07 46 4 7 7 11 Batch Ink Export Performs the Export Ink operation on each sketch in a specified folder directory See Appendix C Exported Ink File Format for a description of the format used for these files 4 7 7 12 Batch History Export Performs the
27. version 3 07 90 sketch saved 1 full path to the file opened new bundle 1 case name assigned to new bundle 2 name of new bundle 3 collections assigned to the bundle new layer 1 ID of containing bundle 2 name of new layer 3 collections assigned to the layer 4 genre 5 pose created none moved 1 dx distance moved along X axis 2 dy distance moved along Y axis rotated 1 angle rotated in degrees resized 1 scale in X direction 2 scale in Y direction 3 anchor point X coordinate 4 anchor point Y coordinate deleted none CogSketch version 3 07 91 changed collections 1 all collections assigned after the change 2 collections that were removed during the change 3 collections that were added during the change changed namestring 1 old name 2 new name changed annotation val 1 old value 2 new value changed relation 1 old relation fact 2 new relation fact 3 all collections assigned after the change 4 collections that were removed during the change 5 collections that were added during the change changed genre pose 1 old genre 2 new genre 3 old pose 4 new pose redrawn none compute positional relations none CogSketch version 3 07 92 analogical comparison 1 tatget ID 2 tatget type note that the base ID and type are given as the object ID and type in the common parameters change grid none at this time mig
28. ANALOGICAL MATCHING 5 3 VISUAL CONCEPTUAL RELATIONSHIPS 5 4 THE KB BROWSER 6 DESIGNING A WORKSHEET ccccccccccccccccccccccccccccccccccccccccccccccccccccsccccccccsccecs CogSketch version 3 07 10 11 11 15 17 19 20 30 35 36 40 48 49 50 51 7 COGSKEICH GRADEBOOK aea Sea ea ea e EU ea Evene NN REN E EEN S E 7 1 ADDING CLASSES ASSIGNMENTS AND STUDENTS TO THE GRADEBOOK 7 2 ADDING SKETCHES TO THE GRADEBOOK 7 3 GRADING WORKSHEETS 7 4 ExPORTING IMPORTING THE GRADEBOOK 8 COGNITIVE MODELING DEMONSTRATIONS ccccccccccccccccccccccccccccccccccccccccccccecs 8 1 PERCEPTUAL SKETCHPAD 8 2 GEOMETRIC ANALOGY 9 DATA GATHERING FOR LABORATORY EXPERIMENTS ccccccccccccccccccccccccccccccccccccccs 9 1 THE SAMPLE EXPERIMENT 9 2 DESIGNING MORE COMPLEX EXPERIMENTS TROUBLESHOOTING dU vevet aues eu a nd des dus 10 1 KNOWN PROBLEMS 11 FUTURE PLANS AND FEEDBACK eR 11 1 WORD BASED CONCEPT PICKER 11 2 AUTHORING ENVIRONMENT FOR SPATIAL ROUTINES 11 3 EVERYDAY PHYSICAL SEMANTICS 11 4 SPECIALIZED INTERFACES amp AUTHORING TOOLS 11 5 SPEECH PROCESSING TZ REFERENCES rnoun 13 APPENDIX A KEYBOARD SHORTCUTS sscccscsscccsscscccsscscscsscccscsscccscssccesccsccess 14 APPENDIX B GLOSSARY 4 S Oe Sea sans vea Or eva e vo ua EA NUS 15 APPENDIX C EXPORTED INK FILE FORMAT scsccscsccscsccscssc
29. Alt S Save As e Ctrl Alt Shift S5 Save sketch as case in the knowledge base e Ctrl F4 Close sketch e Ctrl P Print sketch e Alt F4 Exit Edit operations e Del Delete selected items e Tab Moves to the next glyph upwards in the occlusion order on a given layer and selects it Shift Tab Moves to the next glyph downwards in the occlusion order on a given layer and selects it CogSketch version 3 07 84 View operations Ctrl B Show underlying blob representations Ctrl G Grey out current layer Ctrl R Redraw Ctrl Zoom in Ctrl Zoom out Ctrl W Normal zoom Ctrl Alt W Fit to window Ctrl K KOML server status Ctrl I Inspect knowledge about selected glyphs Analogy operations Ctrl Alt Shift A Compare two layers Help operations F1 Show help CogSketch version 3 07 85 14 APPENDIX B GLOSSARY Collection What represents concepts in Cyc derived knowledge bases Examples of collections include Dog DestructionEvent and SecondOrderCollection Collections are related via monotonic inheritance via the genls relation In CogSketch conceptual labels are typically expressed in terms of collections In open sketching mode the internal collection names are used e g BankBuilding if one drew a bank while in worksheets authors generally provide friendlier everyday names for concepts e g Bank Glyph An entity in a sketch Glyphs consist of a set of 6 the marks which represent the spati
30. CogSketch v3 07 User Manual http www silccenter or Ken Forbus Madeline Usher Andrew Lovett Maria Chang Matthew McLure Subu Kandaswamy Jon Wetzel Kate Lockwood Version of 10 12 2015 Hitchhiker s Guide to the Galaxy Release CogSketch is being developed by the Spatial Intelligence and Learning Center a National Science Foundation Sciences of Learning Center Grant SBE0541957 CONTENTS 1 INTRODUCTION TO COGSKETCH ccccccccccccccccccccccccccccccccccccccccccccccsccccccccccccss 1 1 WHAT IS COGSKETCH 1 2 WHAT YOU CAN DO WITH COGSKETCH 1 3 COGSKETCH OPEN DOMAIN SKETCH UNDERSTANDING Z INSTALLING COGSKETCH vEetrVe e phbaReuves E CE VPERECEN CHE TUR 2 1 SYSTEM REQUIREMENTS 2 2 INSTALLATION 2 3 UNINSTALLING COGSKETCH 3 GETTING STARTED WITH COGSKETCH scecscscscsccscscscscscsccscscscscscsscscscscscseeas 3 1 THE WELCOME SCREEN 3 2 TUTORIALS 3 3 COGSKETCH ESSENTIALS 3 4 WORKSHEET EXAMPLE PARTS OF A CELL 4 THE COGSKETCH INTERFACE CERA ERR ER VR SENTEIN 4 1 COGSKETCH TOOLBAR 4 2 THE LAYER CONTROL PANE 4 3 ADDING ENTITIES TO A SKETCH 4 4 EDITING ITEMS ON THE SKETCH 4 5 LOOKING AT THE KNOWLEDGE IN A SKETCH 4 6 ADDING NEW LAYERS AND SUBSKETCHES TO A SKETCH 47 CoGSkETCH MENUS 5 ADVANCED COGSKETCH FEATURES eee ee eee ene 5 1 SPATIAL REASONING ON SKETCHES 5 2 USING
31. Export Sketch History operation on each sketch in a specified folder directory See Appendix D Exported History Format for a description of the format used for these files 4 7 7 13 Batch History Report Creation Performs the Create History Report action on each sketch in a specified folder and optionally all sketches in subfolders as well 4 7 8 HELP MENU The options on this menu are used for displaying information about CogSketch s current memory usage and for displaying CogSketch s splash screen CogSketch version 3 07 47 5 ADVANCED COGSKETCH FEATURES 5 1 SPATIAL REASONING ON SKETCHES CogSketch performs a variety of visual computations on digital ink including computing qualitative spatial relationships Some of these are automatic and some are done on demand only The automatic calculations include topological descriptions based on the RCC8 relational vocabulary Other qualitative spatial relationships can be computed on demand using the spatial menu The information calculated includes e Positional relationships these are computed between adjacent pairs of glyphs and include 1eftof rightOf above below etc with respect to the glyphs and the corresponding relationships between the entities they describe depending on genre and pose see Section 3 e Relative sizes qualitative and from the set tiny small medium large huge e Articulation points Points that if removed would divide the ink into disconnected
32. Only by representing the parts that make up a glyph can CogSketch compare glyphs identify glyphs that are the same shape and determine the transformations such as rotations and reflections between two glyphs Thus we have been developing a system for decomposing a glyph into its edges and representing those edges in a way that allows two glyphs to be compared via SME the Structure Mapping Engine 3 4 The goal of this subsystem is to build up a flexible hierarchical representation of a sketch that will allow two glyphs or two sketches to be compared at the edge level the object level or even at the level of groups of objects In ongoing and future research we are developing this system as a model of human perception by demonstrating that it produces representations of static two dimensional images that can be used to perform spatial reasoning tasks at human levels e g 14 18 19 This system is accessible through the experimental Perceptual Sketchpad mode A perceptual sketchpad is similar to a regular sketch Like other sketches a saved perceptual sketchpad can be opened with the Open Sketch option from the File menu Users can sketch glyphs within a perceptual sketchpad using the same tools they would normally use The only difference is that a perceptual sketchpad contains a sketch lattice A sketch lattice is a consisting of two more cells with two as the default Glyphs drawn outside of the grid are treated nor
33. The active layer s button is highlighted in yellow pressing a layer s button makes that layer the active layet Each layer button has a few smaller icons that control visibility All layers other than the active one can be made invisible A ghosted layer is grayed out useful on complex drawings when you don t want to be distracted by other layers but still need to see them You can also control whether or not the text labels on a layer are visible orthographic T If you have the Experimenter mode on which can be set from the Preferences menu see Section 0 and have the Debug Layers checkbox selected you will see additional layers for each subsketch representing internal results of the spatial processing For example the Voronoi diagrams and positional relationships can be displayed as overlays This is very helpful for understanding the underlying spatial reasoning that CogSketch is performing The layer control pane also contains a thumbnail of your sketch This thumbnail makes it easier to determine what ink belongs to each glyph The glyphs are color coded In the screenshot to the right you can see that the sun is all one glyph the tree s top is another and the trunk and the ground together are yet another glyph Ink which has not yet been grouped into a glyph shows up as gray 2 6 5 9 in 1 6in inch CogSketch version 3 07 19 The drop down menu at the bottom of the layer control pane allows you to select which su
34. a new glyph or Merge ink into existing glyph gt depending on the nature of your selection if your selection includes all the ink for some existing glyphs you ll be given the choice to merge the selected ink into those glyphs E Properties Ink Properties F E color style 2 4 Glyph Segmentation Split out ink into a new Elyph gt Properties 00000 perties Ink Properties SSS colo styl ll leaves Ly Glyph Segmentation A Merge ink into leaves A Merge ink into trunk trunk 111 ground 4 4 4 GROUPING GLYPHS Grouping glyphs is different than the kinds of merging that can happen when re segmenting glyphs When re segmenting you re just changing what ink belongs to a given glyph If you take all the ink from one glyph and merge it into another via re segmenting you end up with one glyph that has all the ink Grouping allows you to build hierarchies of glyphs where the sub glyphs remain intact Let s say that you have the following sketch of a cat s head the Glyph Map is shown as an inset so you can see the glyph boundaries CogSketch version 3 07 33 eyes mouth head Each of the parts of the cat s head are separate glyphs Sometimes though it s useful to deal with the entire head and one unit To do so select the glyphs and choose Group Glyphs from the Glyphs menu Finish Glyph F7 Import Pictu
35. ae erading report icon To view a grade report select a worksheet and click the View grade report button or right click a worksheet and choose View grade report from the menu that pops up These reports are stored in the same folder that contains the gradebook s copy of the sketches you ve added By default these are added to your Windows Documents folder under cogSketch gradebook organized by class and assignment You can change the location of these sketches if you wish in the Preferences If a grade report is available for a sketch it will be stored in a sub folder with the same name as the sketch e g Documents VCogSketchNeradebookNLayers of the Earth Larkin fot the layers Larkin sketch The report is in HTML format can can be viewed in any web browset If you later decide to change an assigned score just edit the sketch s entry in the gradebook select it and click the Edit button Entering values in the Score fields overrides scores assigned by the automatic grader 7 4 EXPORTING IMPORTING THE GRADEBOOK An instructor will often want to have the contents of their gradebook on multiple computers their computers at the office and at the home for example or you might just want to make a backup copy of the gradebook always a good idea The gradebook provides some tools to make this easy Click on the big Tools button at the left of the gradebook window and choose Export Gradebook Manage Class Sket
36. al aspects of it in the sketch and the content a conceptual entity which is what that ink stands for Ink Processor A software process inside CogSketch that performs some of its visual computations Layer Just as an acetate layer can be laid over a map and drawn upon without changing anything on the map itself a layer allows new spatially registered information to be added to a sketch without changing what else is drawn there Relationships between layer are not automatically computed for example All the layers in the same subsketch are spatially registered Metalayer A special layer where each subsketch is treated as a glyph Relations can be drawn on the metalayer to express relationships between subsketches For example subsketches might represent distinct states of a multi state process with arrows used to explain the sequence in which these states take place The genre of the metalayer is abstract i e the position of two glyphs on the metalayer tells you nothing about the relative positions of the entities in the two subsketches Microtheory A local context within either CogSketch s working memory or the knowledge base Microtheories enable multiple incompatible states to be represented without causing clashes Microtheories are also connected via an inheritance relationship gen1Mt forming a logical environment within which reasoning is performed In CogSketch there is a microtheory for each sketch for each subsketch of
37. any people today speech recognition 1s too hatd to use That said being able to do simple speech commands e g end would enable someone to leave their pen where it is when using a Tablet PC And the technology is improving We will reactivate speech recognition as an option at some point but it is currently very low on out list of priorities Assuming you re a military person who already knows that symbology of course CogSketch version 3 07 80 12 REFERENCES k 10 11 12 Barker K and Blythe J and Borchardt G and Chaudhri V and Clark P and Cohen P and Fitzgerald J and Forbus K and Gil Y and Katz B and Kim J and King G and Mishra S and Morrison C and Murray K and Otstott C and Porter B and Schrag R and Uribe T and Usher J and Yeh P A knowledge acquisition tool for course of action analysis in Proceedings of the Innovative Applications of Artificial Intelligence Conference 2003 Evans T 1968 A program for the solution of geometric analogy intelligence test questions In M Minsky Ed Semantic Information Processing Cambridge MA MIT Press Falkenhainer B Forbus K amp Gentner D 1986 The Structure Mapping Engine Tech Rep No UIUCDCS R 86 1275 UILU ENG 86 1732 Urbana Illinois University of Illinois at Urbana Champaign Department of Computer Science Forbus K Ferguson R amp Gentner D 1994 Incremental structure mapping Proceedings of
38. asking you to name the new subsketch 4 6 5 CLONING A SUBSKETCH Cloning a subsketch allows you to create a new subsketch with the same sketch components as another one This is useful for doing things like drawing a series of steps in a process where you need the same items in each step To clone a subsketch e Switch to the metalayer in the layer control pancl IEEE Clone Subsketch State 2 Delete Subsketch State 2 Paste e Right click on the subsketch that you would like to clone e Select Clone Subsketch from the menu Undo changed glyph properties Show Knowledge for State 2 Edit e You wil be asked to name the new subsketch Properties for State 2 e new subsketch will then open in the CogSketch window CogSketch version 3 07 38 4 6 6 SUBSKETCH PROPERTIES To change the name of a subsketch or one of its other properties select Subsketch Properties from the Edit menu or right click a metalayer glyph while viewing the metalayer and pick Subsketch Properties You ll see a dialog like the following Subsketch Properties teacher bundle Subsketches are used to represent something else for example a subsketch can be used to represent an event or a physical object Select the things your subsketch represents in the yellow box to the right and use the arrow button above to add them to the list of things represented by this subsketch the white box above Things represente
39. being annotated in this case the tree s leaves CogSketch now knows that the temperature of the tree s leaves is 92 F An annotation can apply to more than one item When you hover over a selected annotation glyph a green plus icon appears over its center Click this to add more drag handles In the following figure we ve added the tree s trunk as one of the items annotated Now CogSketch knows that both the tree s leaves and the tree s trunk have a temperature of 92 F You can remove annotation handles by dragging them to the trash icon CogSketch version 3 07 27 What is this v Temperaturelndicator Value 92 degree Fahrenhei NS This is the collection of sketch annotations used to indicate the J J temperature of an object sun leaves Temperaturelndicator END Temperatureindicator Temperat reIndicator iv applies to leaves trunk ground Ink Properties 1 NT Ss mM color style 2ps 4 m 4 t Note that not all annotations allow you to set a quantitative value For example the RotationOrigin annotation is used to indicate the point around which an object can rotate There are no values you can set for this its position on the sketch conveys the necessary information You must draw some kinds of annotation as arrows The position and direction of the arrow you draw gives some of the information These should be drawn as described section 0
40. below E General Preferences L5 Graphics And Sound 5 Server 5 Experimenter 5 Tutor Mouse Drag Tolerance Number of pixels the pen or mouse must be moved while pressed before it registers as a dragging gesture This is used for example to initiate drawing Selection Style In mast Windows applications you add to selections by clicking while holding down the shift key and subtract from them by clicking while holding down the ctrl key This doesn t work well on tablet computers since you often don t have easy access toa keyboard So by default CogSketch uses a different method of handling selections tapping a glyph toggles whether or not itis selected The option below controls whether or not to use this different method for selecting Tap toggles selection Enable tap select in drawing tools Enable Press and Hold for right click pen enabled PCs only Enable Natural Language Concept Entry Always enable draw straight lines tool Phone Home Send us your sketches for science Of particular interest to researchers might be the Experimenter section which among other preferences allows Experimenter Mode to be enabled This makes extra menu options available and allows debug layers like the Voronoi Diagram and Positional Relations to be viewed The Developer mode provides even more information and menu options For students and participants in experiments these should both be turned o
41. bject 754 Foot wall Object 756 Movement b Object 773 Movement a Object 775 displacement Object 839 O image CogSketch version 3 07 Enabledforthisiti Tolerance 0 4 Tutoring Advice Your drawing af this marker bed isn t quite right Enable positian specific advice above Your drawing of Marker bed b is too high Enable positian specific advice below Your drawing of Marker bed b is too low Enable position specific advice left of Your drawing of Marker bed b is too far to the left Enable position specific advice right of Your drawing of Marker bed b is too far to the right 57 The Secutity tab see screenshot below is used to lock your worksheet so that students cannot view the solution Assign a password and click the Apply button If a password has already been assigned and you wish to change it as in the screenshot below the Old Password field will be available you must correctly enter the old password before you can enter a new one You can password protect the solution and the worksheet property editor to prevent unauthorized users from viewing or changing that information Note that you must click the Apply button for your password changes to go into effect Old Password Password eessesesese Re enter Password When a password protected worksheet is opened before you can view or edit the solution you must first choose Unlock Solution from the Edi
42. bsketch is currently active Changing the active subsketch will change the available layers The metalayer button toggles the display of the metalayer The metalayer is a special kind of layer where all of the subsketches are treated as glyphs Thus the same facilities for adding information to glyphs and using arrows to indicate relationships with them see section 4 4 can be used for providing information about subsketches For example you might use the metalayer to add arrows indicating the sequential relationship between steps in a process or to indicate causality The metalayer also provides an excellent way of getting an overview of your sketch 4 2 1 SUBSKETCHES What is a subsketch The observation is that when people are sketching with each other to communicate ideas they naturally carve up a complex sketch into subsketches local neighborhoods within which there is a common framework or point of view In human to human sketching subsketch boundaries are inferred based on the shared world knowledge of the participants and a variety of subtle cues like visual grouping and speech acts The state of the art in vision speech recognition and dialogue management are currently not to the point where automatic subsketch segmentation can be done reliably enough to be unobtrusive Consequently we organize CogSketch s interface to make it easy to create subsketches and switch between them A simple sketch might consist of just a single
43. by decomposing each glyph into edges in order to identify attributes describing the glyph s shape such as straight edged vs curved edged CogSketch will produce a representation for each cell that combines this shape information with the spatial relations between the glyphs Then the representations of the two cells will be compared to identify corresponding glyphs Finally the glyphs will be color coded to indicate which glyphs correspond In the next two sections we first describe how you can interact with two example perceptual sketchpads We then describe how to create and test new perceptual sketchpads 8 1 1 PERCEPTUAL SKETCHPAD EXAMPLES To see the first example sketchpad choose Open Sketch from the file menu and find the sketch PSketchpad_Example1 sk located in the perceptual samples directory You should see a grid consisting of two cells with an shape in each cell Before you can compare the two glyphs let s see what edges CogSketch can find To do this hold down the right mouse button anywhere on the grid outside of the two glyphs and choose Perception Decompose All Glyphs The decomposition process should take only a few seconds although it may require more time on slower machines After decomposition is completed CogSketch will assign a different color to each edge so that you can see the edges it found If you are unclear as to exactly where one edge stops and another begins you can click on an e
44. ce by using the metalayer The metalayer treats each subsketch as a glyph One can move them around and use glyphs to indicate relationships between them We call such descriptions comic graphs because they are similar to comic strips each subsketch corresponds to a panel in the comic strip but they are more general because they can have more complicated relationships than just linearly unfolding Comic graphs are useful for depicting states in a sequence or process CogSketch version 3 07 8 2 INSTALLING COGSKETCH 2 1 SYSTEM REQUIREMENTS e Hardware Tablet PC preferred but not required If you are using a desktop or a traditional notebook we highly recommend adding an inexpensive Wacom tablet because many people find it easier to draw with a pen than with a mouse Operating system Microsoft Windows Vista Windows 7 Windows 8 e CPU At least 1 5 Ghz 2 Ghz or better preferred e RAM 1GB minimum 2 GB or more preferred e Disk Space At least 1 5 GB free disk space is required Please do not redistribute this release since we need to gather information on who is using the software and we want everyone to be using the latest version 2 2 INSTALLATION Run the program named CogSketch v 64bit setup exe or CogSketch v 32bit setup exe that you have downloaded from out web site You will then be guided through the installation process After you have answered the installer s questions CogSketch will be installed
45. ces making sure not to click on an actual glyph and select Do Routine Geometric Analogy This will cause the model to begin building up a representation of each figure in the problem The model will also segment each glyph into its component edges This is necessary so it can determine which glyphs have the same shape as well as computing spatial transformations between shapes e g rotations and reflections CogSketch will also identify qualitative spatial relations between the glyphs in each figure CogSketch will build up a shape representation to describe the edges in a glyph and the relations between those edges as well as a picture representation to describe the glyphs in a figure and the relations between those glyphs After doing this the model will perform the geometric analogy task as described in 20 22 Because the current model is fully implemented in Spatial Routines we will be able to inspect the various operations used to solve a problem When the task is completed CogSketch will highlight the correct answer You can now examine how the model solved the problem Try right clicking inside the upper left square making sure not to click on an actual glyph and choosing Inspect Spatial Routine This will bring up the Routine Inspector which shows the various spatial operations used It will only show you operations that involve the particular square you right clicked on In this case we ate looking at operations
46. ch removes a lot of the delays when inking on a pen enabled computer Otherwise Windows waits until the mouse pen has moved certain distance before passing events to CogSketch just in case you might be making a PenFlicks gesture which means that there is an annoying delay whenever you start to draw Furthermore in the CogSketch preferences you can disable press and hold for right click which reduces the delay even more However due to an annoyance in Windows unless you also disable press and hold for all applications there is still a little delay If you wish to get rid of this delay entirely do the following 1 Open the Control Panel 2 Pick Hardware and Sound Pen and Input Devices or Pen and Touch 3 This opens the dialog shown below On the Pen Options tab click Press and hold and then click the Settings button CogSketch version 3 07 76 Pen actions Use the pen to interact with items on the screen You can adjust the settings for each pen action Equivalent mouse action Single click Double click Press and hold Right click Start Tablet PC Input Panel Pen buttons Use the pen button as a right dick equivalent Use the top of the pen to erase ink where available You can change the amount of time you must press and hold before you can perform a right click equivalent Speed Short Long Duration You can change the amount of time during press and hold to perform a right click action Du
47. ch with them The philosophy of telling CogSketch what you mean is also used when dividing a sketch into pieces and setting up the context for a sketch Sketches are made up of subsketches corresponding to parts of a sketch that are qualitatively similar For example a map might contain an overview of a region and a close up of the street network within a town in that region Similarly a sketch illustrating how a multi step process works might consist of multiple subsketches each subsketch depicting a step in the process CogSketch requires you to identify when you want to work with a new subsketch Subsketches consist in turn of a set of ayers depicting some relevant subset of the information in the subsketch For example CogSketch under some conditions will add a layer to a subsketch illustrating its reasoning in response to a query so that the original data sketched by the user is undisturbed on another layer Layers are modeled after layers in graphics programs which in turn are modeled on the use of acetate sheets over physical papers to easily add and remove spatially registered information when sketching This is not just a collection of terms The meaning of these concepts and relationships are constrained by 1 36 million facts that are also in the KB These are mostly from the OpenCyc knowledge base www opencyc org If you want to learn more about the KB contents we strongly recommend working through the tutorials on
48. ches Se Tools ia iai Preferences CogSketch version 3 07 62 This will create a zip file containing the contents of the gradebook database plus all of its sketches and grade reports To import such a zip file choose the Import Gradebook tool and select the zip file you wish to import Attempts will be made to minimize duplicate entries in your existing database using the imported data if it is newer than what 1s in your existing gradebook CogSketch version 3 07 63 amp CoGNITIVE MODELING DEMONSTRATIONS One goal of CogSketch is to develop models of human spatial cognition Presently CogSketch includes two demonstrations of cognitive modeling the Perceptual Sketchpad and Geometric Analogy Both are implemented using Spatial Routines a modeling approach in which the developer selects a set of basic operations such as encoding an image or comparing two images to identify their differences These operations can be combined to produce a variety of routines for solving different tasks The Perceptual Sketchpad shows how shape information can be combined with spatial information when comparing two images Geometric Analogy shows how this approach can be used to perform visual problem solving We describe the two demonstrations below 8 1 PERCEPTUAL SKETCHPAD CogSketch represents spatial relations between objects in a sketch However some tasks require representing relations between the parts of an individual object or glyph
49. d by the subsketch can be removed using the X button Pushing the More Choices button will give you a wider range of choices for the types of things your subsketch can represent A specialization of Situation Each instance of StaticSituation is a state of affairs between two or more things persisting statically over some time interval Instances of StaticSituation always have a temporal extent and they usually have a tangible and spatial extent As an example consider the situation of Bill Clinton sitting in his easy chair on the evening of 7 4 96 There are participant objects such as Bill and the chair there are relationships such as the seat of the chair supporting his bottom and his weight being off his feet etc In any StaticSituation for the participants in that situation there is some significant or focal relationship between them which does not change In the most typical You can use this to change the name of a subsketch and to tell CogSketch what the subsketch represents 4 6 7 DELETING SUBSKETCHES This can be done from the Metalayer Right click on a glyph that represents the subsketch you want to delete and choose Delete Subsketch from the pop up menu sketches must have at least on subsketch so CogSketch will not let you delete a subsketch if it s the only one in the sketch CogSketch version 3 07 39 4 7 CocSkETCH MENUS E CogSketch File Edit View Glyphs Windows Help Open Sk
50. dge to select it Note that while I shape on the left appears to have been represented perfectly the horizontal edges of the shape on the right have been broken up every place a vettical hash mark intersects them However for many tasks including the one we are about to attempt the system will group the horizontal edge segments back together to form complete edges Once you are done viewing the edges you may go back to the normal view by holding the right mouse button down on the grid and choosing Perception Unmark All Glyphs Remember you must hold the mouse outside of any glyphs to do this To compare the two glyphs hold the right mouse button down on the grid and choose Do Routine Compare Images This process may take several seconds After comparison is complete CogSketch will color the edges to identify the edges that correspond in the two glyphs As you can see CogSketch has identified a set of corresponding edges in which the tight glyph has been rotated 90 degrees from the orientation of the left glyph Because the comparison was only looking at the overall shape of the glyphs the inner hash marks on the right glyph were ignored during comparison They are now grayed out to indicate that no cotrespondence was found between them and edges in the left glyph If you hold the right mouse button down and choose Inspect Spatial Routine you can see the primary spatial Operations that were used by t
51. e DNA as individual glyphs You will get the best results if the cell membrane and the nucleus are both closed curves without any gaps Label your glyphs with the appropriate concepts The previous example showed how to assign concepts using the more complicated concept picker Worksheet authors decide which concepts are available so you won t have to choose concepts from the entire knowledge base Instead once you select a glyph the properties tab in the sidebar shows a list of available concepts and you just select the one you want E CellDNA Workspace finish F nm FN 0 0 ita AU v X color style 4 3 D glyphs ink glyph E Cell e gt Properties Problem Feedback apn Type erw Meta Layer Cell membrane Cell nucleus Chloroplast Cytoplasm molecule Mitochondrion The part of a EukaryaticCell that contains most of its genetic material 11 molecule Uno Wo Ink Properties Cell nucleus M colar style 4 ill pu Est Eon N a Cell membrane Current Subsketch 2 4 4 4in x zoom 9595 4 CogSketch version 3 07 15 Once you have finished your sketch click on the Feedback tab on the sidebar Then click the Update button The tutor will compare your sketch to the teacher s solution sketch and will generate feedback based on the compatison In this case there will be one
52. e any of the VCR relationships check the retract box underneath that relationship and click the submit button You should get the drop down menu again and you can re assign the relationship between the involved glyphs Some things to know about this computation e The set of possible candidates can be quite large or quite small depending on the visual and conceptual properties of the entities involved Having over one hundred possibilities is not uncommon Sorting through them can be complex and we recommend having a KB Browser window open on the side while you are familiarizing yourself with the options e Only relations whose argument type constraints are consistent with the declared types of the entities are presented as possibilities No other information is currently used to filter the set e g it does not try to derive each of the possibilities and its negation using the information in the sketch plus the KB knowledge e The only visual relationships between glyphs that currently trigger this computation are touching and insideness 1 RCC8 EC RCC8 PO RCC8 TPP RCC8 NTPP If you do not get possible relationships for a pair of glyphs check to see if one of these visual relationships holds and if not redraw or move one of them e If the set of relationship options does not include something that you think should be there check to see if that relationship is consistent with the types of entities involved using the KB Browse
53. e formally represented using predicate calculus One of the design tradeoffs in CogSketch is flexibility versus ease of use in selecting concepts As you will see when setting up a worksheet one can provide very easy and natural access to a small number of concepts When sketching in a more open ended fashion one can access a large number of concepts 59 312 to be exact and 15 192 For more information about the nuSketch architecture please see 5 7 9 11 The single best summary of CogSketch 1s 13 See 1 24 for examples where nuSketch systems have been used as components in field tests of larger systems CogSketch version 3 07 6 relationships and 3 652 functions as of 8 1 2011 The current interface when operating in open ended mode shows you the raw concepts This can be a little daunting unless you are an AI expert We are working on several different ways to more naturally get to the concept you want but those are not in this release For example we think that providing natural language phrases is a good candidate if it can be made sufficiently robust We have included a special feature to gather data about what will be required for this the open text concept labeler described in sections 4 3 and 4 4 the sections that discuss adding entities and relations With the open text concept labeler you can just type in an arbitrary piece of text This 1s very flexible but the current version of CogSketch won t do mu
54. eir edges See Section 8 1 for more information on this type of sketches 4 7 1 5 Save Sketch Use this to save a sketch to disk You will only be asked what you want to name the file if the sketch has never been saved to disk 4 7 1 6 Save Sketch As This can also be used to save a sketch to disk You will always be asked what you want to name the file 4 7 1 7 Print Sketch These menu choices are used to send the sketch to a printer Only those layers that are currently visible will appear on the printout 4 7 1 8 Export Sketch as Image Saves a bitmap file to disk with only the currently visible layers shown 4 7 1 9 Exit Closes CogSketch If you have made changes to sketches and have not saved your work you will be asked if you wish to do so as CogSketch shuts down The bottom of the menu shows the four most recently worked on sketches for easy access 4 7 2 EDIT MENU 4 7 2 1 Undo Redo Use this to undo your last action or redo the last action that was undone 4 7 2 2 Copy Paste These work exactly like copy paste in any other application 4 7 2 3 Delete Selected Items Permanently removes all of the items currently selected Items can be selected with the Selection Tool 4 7 2 4 Select Items Activates the Selection Tools See the section on Drawing Tools for more information 4 7 2 5 Unselect All Items Unselects the currently selected items 4 7 2 6 Clear Current Layer Deletes all ite
55. embrane and the second says that the DNA 1s inside the nucleus If you click on an important fact you can enter advice text that will be displayed if the tutor thinks the student s sketch does not meet the requirements of the fact a Facts Important for Tutoring Glyph of Cell nucleus is contained by glyph of Cell membrane Glyph of DNA molecule is contained by glyph of Cell nucleus x 111 What tutoring advice should be associated with this fact You may want to reconsider the location of the DMA molecule In this example if the student does not draw the DNA inside the cell nucleus the suggestion You may want to reconsider the location of the DNA will be given to the student If the fact included quantities e g an annotation indicating that the length of another entity is 800 meters you will also be given the change to specify a minimum and maximum allowed value along with text that will be displayed if the student s sketch falls outside those bounds If the fact is about the orientation of an arrow you can specify an angular tolerance for that arrow For instance if a solution arrow is pointed toward the upper right quadrant then its orientation is 45 degrees Specifying an angular tolerance of 10 degrees will mean that an arrow that is tilted between 35 and 55 degrees will be considered acceptable In interpreting the orientations of arrows degrees means that the arrow is pointed to the ri
56. ertext Ax Analogy Results Jog Base Mapping 33 score 0 092 v Target Virus Insertion Glyph of victim contains Glyph of victim contains Virus Invasion glyph group of a glyph of glyph group of a glyph of Lysosome nucleus Prelude 0 016 Prelude Glyph of victim 0 012 Glyph of victim 7 ee E Glyph of invader 0 008 Glyph of invader Glyph of nucleus 0 008 Glyph of nucleus invader 0 004 invader physical view sketch 0 004 physical view sketch Show Raw Form looking from top sketch 0 004 looking from top sketch Browse SME CogSketch version 3 07 49 5 3 VISUAL CONCEPTUAL RELATIONSHIPS For many reasoning tasks it is helpful to clarify for CogSketch what relationship holds between the objects depicted based on the visual relationships that hold between the glyphs that depict them We call these visual conceptual relationships since they are derived from both kinds of information For example two glyphs that touch could indicate that they are rigidly connected hinged or any of a number of other possibilities depending on the specific kinds of entities involved To specify these relationships for a sketch click on the icon A browser window will open providing a form which enables you to choose a specific relationship for each pair of entities that have a form of visual relationship When you are finished selecting relationships click the submit button If you need to chang
57. etch version 3 07 37 e Unspecified viewpoint This is the only option available for the abstract and discrete graph view and is not available for the geospatial or physical views This pose indicates that nothing can be inferred about the spatial relationships of the contents of glyphs based on the spatial relationships between their ink e Looking from bottom Only available in geospatial and physical views The up vertical direction is considered to be from the user into the sketch e Looking from side Only available in geospatial and physical views The up vertical direction is considered to be aligned with up in glyph space e Looking from top Only available in geospatial and physical views The up vertical direction is considered to be from the sketch to the user Looking from another object Only available in geospatial and physical views No assumption about the orientation of the vertical is made 4 6 3 DELETING LAYERS To delete a layer right click on the Layer Control Pane button associated with the layer you wish to delete and choose Delete Layer from the menu You will be asked to confirm the deletion If you agree to the confirmation then the layer will be removed permanently Note that you cannot delete the last layer in a subsketch since all subsketches must have at least one layer 4 6 4 ADDING A NEW SUBSKETCH Press the New Subsketch button in the layer control panel and you will be presented with a dialog
58. etch Ctrl New Sketch Ctrl N New Design Sketch Ctrl Shift D Save Sketch Ces 4 Save Sketch As Ctrl Alt Print Sketch Ctrl P Close Sketch Ctrl F4 Export Sketch as Image pen Gradebook Exit Alt Fa 1 CA demo sketches db demo spring button skd CN develtree sun sk 3 C _devel house tree bird sk 4 CA V develdiane and bill skcw Current Subsketch State 1 The menus in CogSketch vary according to different types of sketches and configuration settings Students using worksheets and experiment subjects see a simplified selection of menu options while worksheet designers for example see a much larger set of menu choices In this section we ll describe what these menu choices do 4 7 1 FILE MENU 4 7 1 1 Open Sketch Opens a sketch that has been saved to disk This can be used to open any kind of sketch recognized by CogSketch worksheets sKEA sketches and perceptual sketches 4 7 1 2 New Sketch Starts a new general purpose sketch 4 7 1 3 New Worksheet Use this to create a new worksheet Worksheets can be used by researchers performing experiments and by teachers presenting tasks to students If the worksheet has a solution subsketch the CogSketch Tutor can be used to give suggestions to students about their sketches CogSketch version 3 07 40 4 7 1 4 New Perceptual Sketch Creates a new perceptual sketch a special kind of sketch in which glyphs are decomposed into th
59. etch annotations Section 6 describes how to define a worksheet in CogSketch either for classroom use or for gathering data in laboratory experiments Section 7 walks through two demonstrations of cognitive modeling in CogSketch the perceptual sketchpad in which objects are decomposed into their component edges during comparison and geometric analogy Section 8 provides some SILC is a National Science Foundation Sciences of Learning Center CogSketch version 3 07 4 troubleshooting tips Section 9 discusses some of our future plans and how to provide feedback 1 2 WHAT YOU CAN DO WITH COGSKETCH What can you do with CogSketch There are different ways to answer this question depending on what you ate interested in 1 Im an researcher interested in sketch understanding spatial reasoning or tasks that involve sketching CogSketch can be used directly for a variety of computational experiments by using the query and KB extensions built into the software and through the socket based KQML interface For example ancestors of CogSketch were used as interfaces and reasoning components in integrated systems built by Cycorp SRI and the US Army Battle Command Battle Lab Moreover the opt in distributed data collection system built into CogSketch is enabling us to gather a corpus of conceptually labeled sketches that should be a resource for the sketch understanding community a cognitive scientist interested in modeling h
60. etch what relation the glyph represents it tries to figure out the head and tail of the arrow Our recognition algorithm gets confused it you draw lines on the tail of the arrow but performs decently in most other cases If CogSketch could not figure out what direction your arrow is pointing a yellow warning icon will appear over the glyph right clicking the glyph gives you the option to view the problems alerts associated with the glyph If CogSketch s arrow interpretation algorithm determined a direction a thin yellow arrow is drawn over your arrow so that you can see that CogSketch thinks the arrow is pointing in the direction you intended The figure on the right shows positive and negative examples of arrows that CogSketch will recognize CogSketch version 3 07 n Owes money to O lt relation gt Name Object 7884 Ink Properties color style 4px 23 When you first change the glyph type to relation the properties window shows the relation from which you can choose and also puts two handles on the arrow that you drew These handles allow you to select and edit which glyphs the relation is about Ignore them for now Bill n Owes money to relation Name Object 7884 Ink Properties color style 4 px In the box labelled What is this select the relation you want In this example we ll pick Does business with Once you ve chosen a relation CogSke
61. experiments It gatherings timing data for all the points sampled during each ink stroke and for each action the user takes undo redo deletions resizing moving editing labels etc The timing is accurate to the thousandth of a second We don t yet have a full authoring environment for experiments but we do provide a sample experiment that you can edit We can help design more complex experiments Just email us at cogsketch feedback cs northwestern edu More information about the capabilities of an experiment sketch are described later in this section 9 1 THE SAMPLE EXPERIMENT The sample experiment can be found in the following folder Documents CogSketch samples experiment samples Upon opening the sketch you will be asked if you want to run the experiment or edit the sketch e a This sketch contains an experiment Would you like F torunthe experiment or just edit the sketch Edit om 1 sample experiment Task 1 rom f sample experiment Subject Name A 25 cnc Start draw If you choose Run you will be prompted for the experiment participant s ID This can be anything It will be used when saving the sketch containing the participant s work during the experiment Then click the Start button to begin the experiment Your participant will then be presented with the first subsketch in the experiment along with the instructions for that subsketch if any If there are multiple subsketches eac
62. f DNA molecule is inside of the EE d gzlyph group with glyph of Cell nucleus the Cell nucleus Object 7333 container and and glyph of DNA molecule DNA molecule Object 7334 the thing s inside molecule is DNA molecule Glyph zroup with glyph of Cell nucleus the container and and glyph of DNA molecule the thing s inside is a ContainedGlyphGroup Glyph of molecule is contained glyph of Cell nucleus Facts Important for Tutoring Glyph of Cell nucleus is contained by glyph of Cell membrane Glyph of DNA molecule is contained by glyph of Cell nucleus Click the Update button This will populate the Solution Glyphs box with all the glyphs you drew on the solution subsketch Clicking on a glyph in this box will show facts known about that glyph Internal bookkeeping facts and other such facts deemed not useful in tutoring advice are not displayed You must tell the CogSketch Tutor which facts are actually important To do this select a fact and click the button with the picture of an arrow pointing down this will add the fact to the Facts Important for Tutoring section You can use the button with the X to remove facts from this section In the picture above you ll see that two facts about glyphs being contained were marked as important for the tutor The CogSketch version 3 07 55 first one shown says that the cell nucleus is inside the cell m
63. ff Researchers doing data analyses and cognitive simulations will very likely want to enable Experimenter Mode The server page shows options for setting up a KQML server See the document nuSketch KQML API html in the same folder as this document or in the start menu in the CogSketch program group CogSketch version 3 07 43 4 7 3 VIEW MENU 4 7 5 4 Current Layer Grayed Out Toggles whether or not the current layer should be grayed out This does the same thing as the G button on the Layer Control Pane 4 7 3 2 Refresh Window Forces the window to be redrawn 4 7 3 3 Zoom In Make the Zoom In tool active See the section on Drawing Tools for more information 4 7 3 4 Zoom Out Each time you select this the sketch s magnification will be decreased thus zooming out See the section on Drawing Tools for more information 4 7 3 5 Normal Zoom Returns the sketch to a magnification of 1 0 centered in the current window 4 7 3 6 Fit to Window Adjusts the magnification of the sketch such that all the items on all the visible layers will fit inside the sketch s window 4 7 3 7 Visual Conceptual Relation Questions This opens the visual conceptual relations questions page described in Section 5 3 When you click this button a browser window will open where you can answer VCR questions for your sketch 4 7 5 8 Query Window Opens a browser window where you can query the knowledge base 4 7 3 9 Sho
64. ght up is 90 degrees left is 180 degrees and down is 270 degrees In general when specifying minimum and maximum advice for arrows when the student s arrow orientation is below the minimum it should be rotated counter clockwise and when the student s arrow orientation is above the maximum it should be rotated clockwise The tutor can also provide advice if the student drew glyphs differently than the cotresponding glyphs in the solution This is controlled through the Solution Quantitative Ink tab In the next two screenshots shown below the teacher wanted the tutor to critique the student if he or she outlined marker beds differently than those of the CogSketch version 3 07 56 solution The teacher did this by first enabling quantitative ink comparison and then entering an acceptable tolerance and tutoring advice text for each marker bed If you are viewing the solution subsketch while doing this an area filled in yellow will show you visually what the tolerance means In the two pictures below as long as the student s drawing for Marker bed b falls within the yellow area the tutor will consider that acceptable Tutoring Advice Quantitative Ink Comparison The tutor can provide feedback ifa student s glyphs look sufficiently different than those in the solution Enable quantitative ink comparison Solution Glyphs Reverse fault Object 374 Marker bed 6 Object 746 Marker bed a Object 748 Hanging wall O
65. glyph toggles whether or nat itis selected The option below controls whether or nat to use this different method for selecting Tap toggles selection Enable tap select in drawing tools CogSketch version 3 07 31 4 4 2 MOVING RESIZING AND ROTATING Once you ve selected one or more items you can use various click and drag maneuvers to move resize and rotate the items as shown in the picture below When dragging the corner hotspots the aspect ratio of the glyph will be preserved unless you hold down the Ctrl key click and drag on the square boxes to resize Nn click and drag on the round handle to rotate While glyphs are selected you can also move them to other layers by dragging them over the corresponding button on the layer controls pane E tree sun Subsketch 1 Wa finish i 4 Y e O Q X glyphs ink glyph ae Yt sl subsketch lawer m s _ 2 leaves TemperatureIndicator CogSketch version 3 07 32 4 4 3 RE SEGMENTING GLYPHS Changing what ink belongs to what glyph is called re segmenting you re changing how the ink is broken up into glyphs You do this using the Ink Select tool Select the ink strokes you want to break out into a new glyph or to merge into an existing one In the sidebar to the right all available segmentation options will be show in the section labelled Glyph Segmentation You will see choices like Split out ink into
66. h time the participant finishes with one of them and clicks the Next button the next subsketch and its instructions will be presented The final subsketch will have a Finish button When that is pressed the participant s sketch will be saved and you will be asked if you want to restart the experiment Of just exit sample experiment Task 1 c fa finish D 10 glyph P gt Problem You have been givena handout describing the process of cell mitosis Please draw a diagram showing the steps described in that handout ay x Finished CogSketch version 3 07 71 If you instead choose to edit the experiment sketch you will see a screen like this zl sample experiment Subsketch 1 gt E fim P3 7 finish FOX P A Ore Li ti 4 i v zh color style 4px gt Properties Problem new new subsketch layer U wn Calibri 14 y Enter instructions to your participants here Current Subsketch Subsketch 1 40 55 2 13 in zoom 100 Fi You can click the right side and edit the instructions that will be given for the current subsketch If you leave it blank no instruction will be given to the participants and that portion of the screen will be hidden when the experiment is running Each subsketch is treated as a different task within the experiment They will be presented to the participants in
67. hain molecules whose components are CogSketch but is useful for other people looking at your Deoxyribonucleotides are in sketch pairwise structure the overall shape of the molecule being that n of a double helix ONAMolecules Design Coach sketches allow you to assign multiple can be usefully represented as concepts to each glyph For example you might want to say that something is a spring and compressed thing Mm Each time you click on a concept in the list it toggles between being selected or not Sots Lee Ink Properties color style 5 px CogSketch version 3 07 21 4 3 2 2 Glyph Properties entities general sketches When wotking on a general sketch not a worksheet the glyph properties dialog allows you to choose from any collection in the entire knowledge base Our knowledge base is built on top of the contents of the OpenCyc KB so if you have used Cycorp s knowledge base before you will already have some familiarity with the available collections Otherwise we advise reviewing the tutorial materials at www opencyc org Since you can pick any concept from the entire knowledge base choosing the concepts is a bit more complicated than it is with the simplified worksheet dialog In the type in box below the prompt Available Collections start typing the name of the desired concept Once you type at least two letters the box below it shows all concepts beginni
68. he Perceptual Sketchpad In this case there is only one Compare Shapes If you select it it will describe the transformation it identified Now try opening the other Perceptual Sketchpad example sketch PSketchpad_Example2 sk also located in the perceptual samples directory As you can see this sketch contains CogSketch version 3 07 65 several glyphs in each cell of the grid If you want you can decompose the glyphs by holding down the right mouse button on the grid and choosing Perception Decompose All Glyphs However if you instead choose Do Routine Compare Images CogSketch will begin the comparison process by decomposing the glyphs At the end of the comparison process CogSketch colors the glyphs to indicate those that correspond to each other in the two cells Those glyphs on the left that fail to correspond to anything are left black Note that CogSketch combined information about each glyph s shape with information about the spatial relations between the glyphs in order to identify the corresponding glyphs in the two cells 8 1 2 CREATING A NEW PERCEPTUAL SKETCHPAD The first thing to remember when creating a new perceptual sketchpad is that the glyph decomposition code is experimental We appreciate any feedback you can provide regarding problems with the glyph decompositions including bug reports see Troubleshooting Section 9 1 and actual sketch files To make a new perceptual sketchpad go to t
69. he basic organization of the data is as follows Each row represents one point that was sampled as the user drew An ink stroke a polyline in most cases is described by the points Each stroke has its own ID All rows with the same Ink ID are describing the points making up a single ink stroke The data is sorted by timestamp and ink stroke so all the points making up a given stroke will be next to each other A glyph is made up of ink The Glyph Objname column gives the internal name used for the glyph really the object represented by the glyph but that distinction only becomes important when you start doing reasoning about the sketch The Glyph Namestring column gives the human readable name assigned by the user Layers contain glyphs The Layer Objname and Layer Namestring columns give the internal name of the layer and the human readable name respectively A subsketch is a collection of layers These correspond to cases microtheories in our reasoning system so each subsketch gets its own reasoning context The Subsketch Case column is the name of the associated microtheory and the Subsketch Namestring is the human readable name for the subsketch Sketches are made of one or more subsketches These also correspond to microtheories each subsketch microtheory inherits from the overall sketch microtheory The Sketch Case column gives the name of sketch s microtheory the Sketch Namestring
70. he corresponding edges Finally there is a Recognize Shape operation for the small dot In this case the model hadn t seen a dot before the model only considers shapes CogSketch version 3 07 68 in the current problem so it is only recognizing that is has seen a shape before in the same problem Let us suppose you ve clicked on Browse Facts You can now see the facts describing the differences between images A and B These facts are similar to the ones described in 20 although the terminology has changed somewhat changeBetweenImages describes a fact that is true in one of the images but not the other In this case we can see that one change between Image 2 and Image 1 is that Image 1 has an object above another object This above object is arbitrarily called Element 3 but you can see from the color coding that it 15 the dot changeBetweenImagesFromTo describes how an object changes from one image to another In this case the change to Element 3 is simply that it is absent in Image 2 but present in Image 1 Finally note that Element 3 s shape type StrictShapeTypeFn 2 0 has also been recorded This shape type is represented by an atbitraty symbol Because the recognize shape operation failed to find other objects with the same shape as the dot it 1s the only object so far with this shape type Later the model will recognize that other images in the problem also contain dots and they will be assigned the same sha
71. he file menu and choose New Perceptual Sketchpad CogSketch will create a new sketch containing a grid with two cells The grid will be sized to fit the current window size when the sketch is first created If you later resize the window you may find the grid to be too large or too small However you can resize the grid by click on any edge of the grid and dragging To draw a glyph in the sketchpad simply follow the usual rules sketch out the glyph and then click on the Finish Glyph button Remember you can hold down the right mouse button and choose Perception Decompose All Glyphs to see what edges CogSketch found in your glyphs If the glyphs have already been decomposed you can also select Perception Show Edges and Junctions for Glyphs Afterwards you can hold down the right mouse button on and choose Unmark All Glyphs to go back to the normal glyph view although this is not necessary Once at least one glyph has been drawn in each cell of the grid you can right click on the grid and choose Do Routine Compare Images to compare the glyphs in the two cells of the grid As explained in the previous section the comparison will work differently depending on whether there is only glyph in each cell or there are multiple glyphs If there is just one in each cell CogSketch will compare the two glyphs and color code the glyphs edges to indicate the corresponding edges If there are multiple glyp
72. hs CogSketch will color code the glyphs themselves to indicate the corresponding glyphs leaving those glyphs that fail to correspond in their original color 8 2 GEOMETRIC ANALOGY To see our geometric analogy model in action start CogSketch and open the sketch GeometricAnalogy sk in the perceptual samples directory This file contains our adaptations of four geometric analogy problems from Evans classic ANALOGY paper 2 To run the full set of 20 problems you can load FullGeometricAnalogy sk Note that this file may take a minute or two to load You can use the file to run the model on those CogSketch version 3 07 66 problems modify the problems or write your own problems from scratch Let s begin by going to Evans problem 1 Make sure 1 is selected under Current Subsketch I GeometricAnalogy 1 o 2 finish 1 elyphs ink color style 4px Q D xf 34 Ww gt Properties LJ new new subsketch layer 4 Select a glyph or ink to edit its properties Meta Layer n m Current Subsketch 1 v 5 5 0 3 in a zoom 43 4 t T This subsketch contains three sketch lattices Sketch lattices are used to break up a sketch into parts In this case the two top lattices describe the problem A is to B as C is to while the bottom lattice describes the five possible answers Hold down the right mouse button on any of the latti
73. ht need to describe what was changed but Pm not sure any experimenter cares The complete details are however recorded internally in the sketch s history change registration none at this time might need to describe what was changed but Pm not sure experimenter cares The complete details are however recorded internally in the sketch s history show vcr questions none answet vct question 1 new fact retract vcr answer 1 old fact tutor suggestions 1 tutor suggestions given as a list of strings CogSketch version 3 07 93 17 INDEX Articulation points 48 Bitmap layer 36 Comic graphs 8 Conceptual labeling 21 how to 12 Drawing glyphs how to 12 Genre 8 Glyph 12 Glyphs moving resizing and rotating 32 Ink of a glyph 12 Ink processor error recovery 19 KB Browser 51 Layers 7 Looking from top 38 Metalayer 8 CogSketch version 3 07 Pose 8 Looking from bottom 38 Looking from side 38 Looking from top 38 Unspecified 38 Positional relationships 48 Qualitative topology 50 Recognition 6 Relative sizes 48 Segmentation 20 Subsketch 7 Clone 38 definition 20 View Abstract view 37 Discrete graph view 37 Geospatial view 37 Physical view 37 94
74. idual ink strokes and the Glyph Select tool when you re editing entire glyphs at a time When this tool is active each of your glyphs will be drawn with a colored CogSketch version 3 07 30 background so you can more easily see what ink belongs to what glyph You select items by drawing a lasso around them or by clicking on them E tree sun Subsketch 1 de a nm finish i 9 glyphs ink glyph zc p g gt At At a Properties I new Lj new subsketch layer Ink Properties Exc color style 4px Glyph Segmentation Split out ink into a new glyph m Current Subsketch Subsketch 1 n 1 8 2 5 in zum 100 4 A a You can also select glyphs when using the Drawing Tool if you enable it in the user preferences it s disabled by default To enable it choose Preferences from the Edit menu and check Enable tap select in drawing tools see picture below Once enabled you can click on glyphs i e tap them to select and unselect them Selection Style In mast Windows applications you add ta selections by clicking while holding down the shift key and subtract from them by clicking while holding down the ctrl key This doesn t work well on tablet computers since you often don t have easy access toa keyboard So by default CogSketch uses a different method of handling selections tapping a
75. ilitate finding worksheets or build your own using the built in Authoring Environment Please note that building your own requires investing time in understanding much more about the ontology and spatial relationships that CogSketch computes which is not required for using pre existing sketch worksheets We are especially interested in feedback on how to improve the authoring process CogSketch version 3 07 5 b Design Coach is aimed at helping engineering design students learn to use sketching to communicate ideas Students explain their designs by drawing a kind of comic strip plus filling out a language like form which describes aspects of a design that are difficult to draw CogSketch can reason qualitatively about forces and surface contacts 2D drawings 26 and this ability is used to provide feedback about the student s explanation 27 29 No matter what your interest in CogSketch is we are interested in hearing from you about what you are using it for and whatever suggestions you might have for improvement cogsketch feedback a cs northwestern edu We would also appreciate it if you opt in to allow your sketches to be sent anonymously to us periodically so that we can add them to the corpus that we are building 1 3 COGSKETCH OPEN DOMAIN SKETCH UNDERSTANDING When people sketch they typically talk to each other about what they are drawing Not everyone is an artist and even artists will often scribble a few suggest
76. ive strokes when working through an idea on their own or when talking an idea through with other people When sketching with another person we don t have to recognize what they are drawing based on its shape We re typically told via language what is being depicted In other words visual recognition is not essential to sketching Recognition can be a catalyst accelerating interaction and making it smoother but it is not a requirement This insight is at the heart of our approach to sketch understanding We have the user indicate what ink belongs to a 0 0 a visual entity that is part of a sketch and use special purpose interfaces to tell CogSketch what they want that glyph to mean So while you have to do a little more work to tell CogSketch what you are drawing you never are blocked in what you ate trying to do by recognition failures Our experience with several prior systems based on these principles is that people can learn the basics of the system very quickly On the other hand CogSketch does visual processing of what is drawn using routines and representations designed as models of human visual processing 21 This has enabled CogSketch to simulate a variety of visual tasks and we are continuing to expand the set of tasks in order to achieve a broadly capable model of visual reasoning When you assign a meaning to a glyph you are selecting a concept or relationship from a large library The elements in this library ar
77. ketch error logs The format of error logs is lt year gt lt month gt lt date gt _errot_ lt id gt log e g 20080111 error 000 log The ID gets incremented each time an error occurs Note that the error logs can get large If a log is over 1 MB or so you may want to zip comptess the file before emailing it since most email providers have a limit on the size of email attachments 5 Any other information you can provide For example a screen capture showing what error dialog popped up or illustrating a display problem is often helpful Another source of information to include in a bug report 1s information from the console window While CogSketch is running a small CogSketch icon will appear in your Windows tool tray Double clicking on this icon will pop up the console window It is important to not close this window while CogSketch is running since that will cause CogSketch to exit The console window is closed automatically upon exiting CogSketch If CogSketch seems unresponsive check the console window and you may find an error message displayed If there 1s please copy and paste this into your bug report If the console window is still responding you can generate a backtrace by typing zoom all t vetbose t Such traces can be very useful to include in your bug report When you hover your pen or mouse over it a tooltip appears saying Allegro Common Lisp Console because CogSketch is written in Allegro Co
78. ketch All Collections aug 2 Ed i EV CellMembrane cell membrane that will be added to the knowledge er E base anytime this sketch worksheet is em used To do this click the Define x new collection button A dialog will The part of a EukaryoticCell that appear that allows you to specity A contains most of its genetic material name and description for your new collection Please adhere to Human readable name for the term CellNucleus collection naming conventions begin cell nucleus with an upp er case letter and use Allow extra glyphs of this type camel case When you define a new Allowed Relations collection it will automatically be added to the Collections Allowed in Sketch The X button between the boxes can be used to remove concepts from the Collections Allowed in Sketch section You wyhat js this can optionally provide a more readable name for a concept Properties Cell membrane by selecting it in the Collections Allowed in Sketch ESI EIE section and entering a name in the box titled Human Chloroplast readable namestring for the term DNA molecule The collection of all long chain When the student or experiment participant labels a glyph molecules whose components are they will see these collections as available options in the oe E pairwise structure the overall sidebar shape af the molecule being that
79. ll be visible at the same time However there can only be one active subsketch at a time Subsketches ate useful for representing distinct pieces of information for example different steps in a process The only way to see all the subsketches in a sketch at once is to use the metalayer Using the metalayer you can define relationships between subsketches such as the order of steps in a process Another example of when you would use multiple layers or the metalayer is when doing analogical reasoning see section 5 2 4 6 1 ADDING A NEW LAYER Press the New Layet button in the layer control panel and you will be presented with the following dialog Layer Name Energy Layer Type Normal Layer Copy the current layer Enter a name for the layer Layer Type can either be Normal Layer or Fixed Image Layer A Fixed Image Layer allows you to include a picture as a background image for your sketch Layer Name Energy Layer Type Image File D _data _Pictures SPACE digital flar CogSketch version 3 07 36 4 6 2 LAYER PROPERTIES If you later want to change the name of a layer or edit one of its other properties select the layer and choose Layer Properties from the Edit menu or right click the layer in the layer control pane and choose Layer Properties from the shortcut menu You will see a dialog like the following Layer Properties teacher layer Layer Name Cell Diagram Layer Type Normal La
80. mally However any time a glyph is drawn inside the grid CogSketch will decompose the lines in the glyph into one or more perceptual edges It will build up an internal representation of each edge s features as well as the spatial relationships between the edges Once the decomposition process has been completed the user can view the system s decomposition of the glyph or use the decomposition to compare the glyph to other glyphs The perceptual sketchpad supports two types of comparisons comparisons between individual glyphs and comparisons between groups of glyphs If there is only a single glyph in each cell of the grid CogSketch will directly compare the two glyphs attempting to identify their corresponding edges If there is a one to one mapping between the edges in the two glyphs it will attempt to identify a transformation between the glyphs such as a rotation or reflection Assuming it finds a valid transformation including a perfect identity CogSketch version 3 07 64 match it will report this transformation and color code the edges of the two glyphs to identify which edges correspond Note that the Perceptual Sketchpad requires a very close match for transformations each of the corresponding edges must be rotated or reflected about the same axis and must show the same change in length If the edges don t match closely enough it will not return a mapping If there are multiple glyphs in the two cells CogSketch will begin
81. mmon Lisp CogSketch version 3 07 74 CogSketch version 3 07 75 10 1 KNOWN PROBLEMS e One person had trouble installing CogSketch on Windows 8 the installer aborted and gave the error message The wizard was interrupted before CogSketch could be completely installed Getting the latest Windows Updates fixed the problem Early versions of Windows 8 apparently did not completely support the Windows Installer API in a backwards compatible mannet e If CogSketch crashes it can lead to corruption in the knowledge base The best way to fix this is to revert to the knowledge base from the original distribution or from a backup that you have made Please see Section 2 2 for instructions e While ink processor crashes have become quite rare they are still possible The healing touch of your cursor on the icon representing the ink processor that is crashed should restore it e Usually if your graphics driver doesn t support CogSketch s OpenGL windows you ll get an error message saying something to that effect On at least one machine configuration though the sketching windows failed to be created but no error message was generated when creating a new sketch or opening an existing one CogSketch hung while displaying a message saying Starting MMI and Ink Processor 7 and no sketching window was created Installing the latest graphics driver for that machine fixed the problem CogSketch disables Windows Pen Flicks whi
82. ms from the current layer CogSketch version 3 07 41 4 7 2 7 Layer Properties Allows you to edit the name genre and pose of the current layer 4 7 2 8 Delete Current Layer Permanently removes the current layer from the sketch 4 7 2 9 Subsketch Properties Allows you to change the name of the current subsketch and to tell CogSketch what the current subsketch represents 4 7 2 10 Worksheet Property Editor This is used by designers of worksheets 4 7 2 11 Sketch Grid Properties You can choose to have a grid displayed in the drawing area This menu option allows you to choose whether or not the grid is visible and to change the spacing line color and line thickness of the grid 4 7 2 12 Change Sketch Registration Use this to register your sketch with real world coordinates You can set it up for example so that each unit of a sketch corresponds to one meter and that the lower left corner 15 Om Om 4 7 2 13 Edit Arrow Directions Allows you to edit the arrowDirectionForRelation facts that are used to tell the ordering of the objects assigned to arguments of a relation expressed by a drawn arrow Please do not attempt to use this until you are very familiar with the OpenCyc ontology and CogSketch representation conventions CogSketch version 3 07 42 4 7 2 14 Preferences This will open the preferences window thus allowing you to change configurable options The general preferences are shown in the picture
83. n subsketch and that subsketch will be selected This new subsketch will include any layers you added to the workspace subsketch and those layers will be created with the same properties 1 name genre and pose that were used in the workspace The Worksheet Property Editor will then show additional options on the Solution tab described below Draw your solution in this subsketch as you would any normal sketch Label the glyphs add new layers as needed etc When you add or delete a layer on the solution subsketch it will automatically be added or deleted on the workspace subsketch Note that this does not go the other way after the solution subsketch has been created layers added deleted on the workspace subsketch are not automatically added deleted on the solution This is intentional It allows you to have different layers in the workspace if you so desire e g to provide an extra layer in the workspace for student intermediate work CogSketch version 3 07 54 After you have drawn the solution go back to the Solution Important Facts tab of the Worksheet Property Editor Include solution subsketch Feedback button enabled Tutaring Advice Facts Important for Tutoring Selecting a glyph in the Solution Glyphs section below will show facts about the glyph that might be useful for tutoring Choose which facts you want to be considered important Solution Items Facts About Selected Item c Glyph o
84. ng one minute after opening it the timestamps of that bit of ink will start at 560 seconds no matter how long the sketch sat around closed So basically timestamps include only the time that the sketch was actually open e Ink Point Timestamp clock time Clock time is given in the ISO 8601 standard with millisecond precision CogSketch version 3 07 89 16 APPENDIX D EXPORTED HISTORY FORMAT The files saved by the Export Sketch History option on the File menu are comma separated value files they contain rows of data where each row represents an action performed and the data within the rows are separated by commas There are several fields columns if you view this in a spreadsheet program common to all types of recorded actions Action Type moved deleted created sketch opened etc Action Subtype start finish e Sketch Time e Clock Time given in ISO 8601 standard Source of action user ink processor etc Source Detail the username when source is user e Sketch Name Subsketch ID Subsketch Name e Layer ID e layer Name e ID of object acted upon e Type of object acted upon Then there are five fields for parameters specific to the given type of action These are labeled Param 1 through Param 5 The various actions use them as follows sketch created 1 case name assigned to the sketch sketch opened 1 full path to the file opened sketch closed none CogSketch
85. ng with those letters Select one and click the green up arrow button to add it to your glyph or just double click the concept that adds it to your glyph without your having to click the green arrow button You can assign more than one concept to a glyph If you need to remove a concept select it in the Assigned Collections box and click the red X button You can use the Name field to enter the text label you want to appear on your sketch It won t mean anything to CogSketch but is useful for other people looking at your sketch CogSketch version 3 07 gt Properties Glyph Type entity m Assigned Concepts Condominium Workplace M Available Concepts Condominium Condolence Condom Condominium Condor An instance of ArtifactTypeByGenericCategory Each instance of Condominium is a ResidentialBuilding whose SingleResidencellnits are individually owned and whose I common areas are jointly owned c Name Object 7699 Ink Properties le 4 rd r 22 4 3 2 3 Glyph Properties relations Worksheets and Design Coach Sketches Initially the basic glyph type for any glyph is entity If instead you want the glyph to represent a relation between two objects change the glyph type to relation Note that this glyph type might not be available if the teacher who designed the worksheet didn t allow any relations Relations must be drawn as arrows Once you tell CogSk
86. nis in UniversalVocabularyMt Glyph in NuSketchMt SpatialT hing specs in BaseKB HugesizeGlyph LargeSizeGlyph MediumSizeGlyph NotVeryRoundGlyph QDiagonalDownwardGlyph QDiagonalUpwardGlyph QHorizontalGlyph QVerticalGlyph SmallSizeGlyph SomewhatRoundGlyph TinySizeGlyph VeryRoundGlyph Knowledge Base C Documents and Settings All Users Application Data aRGiCogSketchikBiOpenCyc KB 51 6 DESIGNING A WORKSHEET This section describes how to develop a new CogSketch worksheet As noted earlier sketch worksheets provide tutoring by comparing an expert s sketch with a student s sketch Developing a new worksheet thus requires specifying what concepts are involved drawing the expert s sketch and providing advice based on potential differences Here we use the Cell DNA worksheet that comes with CogSketch as an illustration To create a new worksheet choose New Worksheet from the File menu A new blank sketch will be created and the Worksheet Property Editor will be opened To edit an existing worksheet simply open the file use Open Sketch from the File menu Choose Worksheet Property Editor from the Edit menu if you want to see the editor Please note that the Worksheet Property Editor menu option is only available if you are in Experimenter Mode or if you are viewing a solution subsketch It is hidden otherwise since we don t want students and research participants to be able to edit the worksheet The sub
87. nt in a sketch 15 called a ghph Glyphs have two parts The glyph s mk represents its visual aspects and is what you draw The content of a glyph is the symbolic representation of what you are depicting You draw a glyph as follows Up You can draw when the drawing tool is selected in the sketching window s toolbar Hold down your mouse s left button and drag to draw or press your pen down on your tablet and drag it Once you ve drawn one or mote ink strokes click the Finish Glyph button This tells CogSketch to group the ink strokes you ve just drawn into a single glyph Any time you press the Finish Glyph button all ink you ve drawn since the last time you pressed that button will be made into a glyph You need to group the ink into meaningful entities and this 1s the method for doing so As an example say you wanted to draw a house a tree and a bird It makes sense to have one glyph for each of these entities so you would draw several ink strokes making the house click Finish Glyph draw the ink for the tree click Finish Glyph and then draw the bird and click Finish Glyph Periodically look at the glyph map the small thumbnail on the left of the screen It uses false colors to show you how you have divided your ink into glyphs If you discover that you need to break up a glyph into more than one glyph or merge multiple glyphs into one glyph you can use the lasso Section 0 to do this You now have three glyphs on your sketch
88. olution subsketch The solution sketch contains grading criteria entered using the Worksheet Property Editor described in the section titled Designing a Worksheet The automatic grader uses the same CogSketch tutor that can give advice to students while they are working on the worksheets plus the grading criteria to assign scores Since the grading criteria is extracted from your designated solution sketch you don t have to decide on it at the time you give the assignment to the students You can change it whenever you wish and of course you can re run the automatic grader at any time To use the grader just select one or more sketches and click the Grade worksheets button The grader will first open the associated solution sketches in CogSketch If the solutions are CogSketch version 3 07 61 passwotd protected you will be prompted at this time to enter the passwords Then each of the student worksheets will be opened in CogSketch and graded Besides assigning a score to each one a grade report will be created for each worksheet that details each of the criteria on which the student was graded These reports can be viewed through the gradebook when one is available for a worksheet an icon will appear beside that worksheet s listing in the gradebook as seen in the screenshot below Title Student s A layers solution solution sketch gt Larkin Patricia Umm Rummler Rummler Daria layers Wright Wright Dont
89. on Anish glyph At that point all the free ink is grouped together into a single glyph In other words CogSketch treats all of the ink drawn between the button presses as one glyph If you later need to change how your glyphs are segmented you can use the ink select tool to select ink strokes and split them out into new glyphs or merge them into existing glyphs 4 3 2 CONCEPTUAL LABELING Conceptual labeling happens after you have finished drawing the glyph Whenever you select an item or multiple items editable properties are displayed in the sidebar to the right of the drawing area If you have a single glyph selected those properties include the conceptual labels The available glyph properties are different for worksheets and regular sketches and they vary with the basic glyph type The gt following sections describe the differences 4 3 2 1 Glyph Properties entities Worksheets and aa Design Coach Sketches What is this The instructor who designed the worksheet determines ERA m a im Cell membrane w ic concepts are availab e so t glyph properties dialog is simplified You given a list of concepts Chloroplast you just select the one you want If the worksheet Cytoplasm designer allows it you will also be given a field called Mitochondrion Name You can use this to enter the text you want to appear on your sketch It won t mean anything to lhecellectien ofall long c
90. on is increased This button also has a pull down menu offering other zoom options 4 1 1 2 E Zoom Out Each time you click this button the sketch s magnification is decreased This button also has a pull down menu offering other zoom options 4 11 3 Drawing Tool This is the tool that s normally active It allows you to draw ink strokes for glyphs and optionally select glyphs by tapping them see Edit Preferences General Depending upon your preference settings or the settings authored into a worksheet this button can have a pull down menu offering other kinds of drawing pens Currently these include the normal freeform drawing tool a straight line tool and a circle tool 41 14 Glyph Select Use this tool to select glyphs on the current layer of the sketch You can either tap to select glyphs or draw lassos around them Selected items may be resized moved and rotated 4 11 5 Ink Select This lasso tool allows you to select ink strokes within glyphs Selected ink strokes can be resized moved and rotated While ink is selected it can also be split out into a new glyph ot merged into existing ones CogSketch version 3 07 17 4 1 1 6 Erase Ink The eraser tool is used to erase parts of ink strokes Holding down the button allows you to select from three different sizes of eraset 4 1 2 OTHER CONTROLS ON THE TOOLBAR finish glyph 4 1 2 1 Finish Glyph Use this button to segment the glyphs in your
91. ow one student subsketch in a worksheet We may extend CogSketch to support multiple student subsketches in the future CogSketch version 3 07 52 The Allowed Collections section on the Workspace Concepts tab 1s where you specify what conceptual labels can be assigned to the glyphs in a sketch Optionally this can be Free Text Entry in which case the student can type whatever they like when labeling a glyph Such free text won t mean anything to CogSketch or its tutor but it may be useful for some experiments If you don t use Free Text Entry you will have two options you can select concepts from the knowledge base or you can define new concepts altogether We provide out knowledge browser click the Browse Knowledge base button for seeking out concepts that are already defined in the knowledge base Once you know what concept s you want start typing its name in the box under All Collections on the right As you type all collections starting with those letters will be shown in the box Z Sketch 3 Cell DNA Worksheet below Select one click the left arrow and it will be added to the Solution quantitativeink Feedback Rubrics Questions Grading Security Collections Allowed in Sketch If e Solution Important Facts you cannot find the concept that you Allowed Collections are looking for in the knowledge F Free Text Entry m b ni d fin n ll ti n Collections Allowed in S
92. p AUTHORING TOOLS We see the worksheet interface as the first of a set of specialized interfaces that allow CogSketch to be customized for particular experimental and or educational purposes Customization can allow us to develop specialized concept picking interfaces that are very CogSketch version 3 07 79 efficient and easy to use For example our military sketch understanding system 11 lets you select from over 800 different concepts easily Our goal 1s to have a library of interface types to make it straightforward for scientists to put together experiments and for curriculum and educational software designers and ideally teachers and students to create new activities and tutoring systems Given feedback on what types are most effective we can create authoring tools that simplify creating specialized interfaces 11 5 SPEECH PROCESSING Given that people talk to each other during sketching why don t we use speech recognition in CogSketch The answer is a little complicated First prior nuSketch systems did support speech recognition We dropped it for CogSketch for several reasons First speech I O is generally unsuitable for classroom situations Second unfortunately today s speech recognition systems are still not very good To get high recognition rates requires the speaker to train the system wear a noise cancelling microphone and carefully restrict the vocabulary and grammar of what is said This means that for m
93. pe type At this point if you want to consider the problem further you could try clicking on Infer Image gt Image D to see how the model inferred the answer image Alternatively you could open up a Routine Inspector for one of the other squares in this problem Feel free to repeat this process for any of the Evans problems contained in the files GeometricAnalogy sk and FullGeometricAnalogy sk You can also create your own variations of the problems by copying and pasting glyphs deleting glyphs use the delete button on the toolbar or the delete key to delete a glyph or moving glyphs around Finally you can go to the final subsketch entitled Make Your Own and build a new geometric analogy problem from scratch There are two ways to make your own glyphs You can either draw them in CogSketch or import them from PowerPoint You can import from PowerPoint by drawing a shape in PowerPoint and then copy and pasting into CogSketch The advantage of using PowerPoint is that you can build exact shapes with perfectly straight lines such as the ones used in this example While our system for decomposing a glyph into its edges works reasonably well on hand drawn glyphs it can make errors Thus if you find the model isn t working well with hand drawn glyphs you can switch to PowerPoint shapes CogSketch version 3 07 69 CogSketch 9 DATA GATHERING FOR LABORATORY EXPERIMENTS CogSketch can be used as a platform for laboratory
94. r e Earlier versions of CogSketch used MAC FAC 12 to retrieve sketches matching on both visual and conceptual properties to provide suggestions for relationships based on similarity to previous sketches 10 This capability is currently disconnected but it may be reactivated depending on how people are using CogSketch CogSketch version 3 07 50 5 4 THE KB BROWSER CogSketch includes access to FIRE s knowledge base browser via the View Browse KB button keyboard shortcut Ctrl K Below is an example of how it looks displaying information about a collection Typing a string in the search box in the upper left will lead to a set of completions being generated when you hit the search button The completions are entities collections and relationships from the underlying knowledge base NuSketchGlyph all genls all specs all instances all references Dynamic Cases e MinimalCaseFn CaseFn CogSketch version 3 07 List All Explicit Cases Edit KB Facts FEM OpenCyc NuSketchGlyph C Documents and Settings All i Users Application ollection Data QRG CogSketch KB comment A NuSketchGlyph is a group of sketch components that can be considered to have a coherent meaning An example would be a group of ink representing an armor battalion comment The collection of all instances of Glyph created using NuSketch isa in UniversalVocabularyMt ObjectType in Nu SketchMt Collection ge
95. r J Lovett A amp Wetzel J 2011 CogSketch Sketch understanding for Cognitive Science Research and for Education Topics in Cognitive Science pp 1 19 14 Jee B Gentner D Forbus K Sageman B and Uttal D 2009 Drawing on I5 16 17 18 19 20 21 22 2 05 24 29 experience Use of sketching to evaluate knowledge of spatial scientific concepts In Proceedings of the 31st Annual Conference of the Cognitive Science Society Amsterdam The Netherlands Kim H 1993 Qualitative reasoning about fluids and mechanics Ph D dissertation and ILS Technical Report Northwestern University Lockwood K Lovett A and Forbus K 2008 Automatic Classification of Containment and Support Spatial Relations in English and Dutch In the Proceedings of Spatial Cognition Lockwood K Lovett A and Forbus K 2008 Automatic Classification of Containment and Support Spatial Relations in English and Dutch In Spatial Cognition VI Learning Reasoning and Talking about Space Lecture Notes in Computer Science 5248 pp 283 294 Berlin Heidelberg Springer Verlag Lovett A Gentner D amp Forbus K 2006 Simulating time course phenomena in perceptual similarity via incremental encoding Proceedings of the Twenty Eighth Annual Meeting of the Cognitive Science Society Lovett A Forbus K amp Usher J 2007 Analogy with qualitative spatial representations can simulate sol
96. ration Short Long Test settings To test your settings press and hold the graphic The graphic changes when you perform press and hold successfully CogSketch uses OpenGL for rendering On some very old Tablet PCs OpenGL hardware support only works in landscape mode not portrait mode If you start CogSketch in one display mode and switch to the other it will hang on such machines CogSketch version 3 07 77 The only workaround for such machines is to close CogSketch before switching display modes CogSketch version 3 07 78 11 FUTURE PLANS AND FEEDBACK We have a number of planned extensions for CogSketch We seek your feedback to help us prioritize Here are a few of the things we are thinking of 11 1 WORD BASED CONCEPT PICKER A high priority for us is making it easier to find concepts when authoring Sketch Worksheets and in the open interface mode Open interface mode is designed for scientists and developers so we don t feel the need to shield you from the KB internals once you ve found them But we would like to make it a lot easier for you to find and extend them The KB already contains mappings from WordNet to OpenCyc concepts in it We plan to exploit these mappings so that you can enter a word into the concept picker box and get back a small set of choices of possible concepts to use The free text concept labeler is a way for us to gather data on the vocabulary and phrasing that people tend to use
97. re as Glyph Group Glyphs Same Objects A glyph will be created in which the selected glyphs are sub glyphs All the facts about the sub glyphs remain intact so CogSketch still knows the head 15 a Head AnimalBodyPart etc but they re been grouped together to form a new single entity You can give the new glyph a name conceptual labels and such just like you would with any other glyph If you want to edit the sub glyphs or you just want to get rid of the grouping just select the group glyph and choose Ungroup Glyphs from the Glyphs menu 4 4 5 EDITING OTHER GLYPH PROPERTIES If you decide to change the name of an entity the collections assigned details about a relation glyph or annotation glyph etc you just do it the same way you initially assigned those properties as described in section 4 3 2 Conceptual Labeling In other words select the glyph and use the sidebar on the right CogSketch version 3 07 34 4 5 LOOKING AT THE KNOWLEDGE IN A SKETCH CogSketch maintains lot of information for sketches based on both the visual information in the sketch and the knowledge from the Cyc knowledge base Operations performed on a sketch for example calculating spatial relationships also contribute to this knowledge To view the knowledge you can either choose the Show Knowledge about Sketch option under the View menu or the Show Knowledge about Selected Item keyboard shortcut ctrl D also available on
98. represent the number of tasks that processor has left to do CogSketch version 3 07 18 Idle the spatial processors are waiting for new tasks Processing the spatial processors are actively wotking on some of the ink drawn on the sketch The left eye indicates the status of the main ink processor The right eye indicates the status of the Voronoi processor eu Disabled the spatial processors are disabled during certain interactions like drawing and moving glyphs to improve the responsiveness of the system There is currently a minor bug wherein the processors can get stuck in this state If this status persists even if you re not in the glyph drawing mode or moving resizing or rotating a glyph then the system is slacking Clicking the eyes will restart the spatial processors Error the spatial processors encountered an error and are no longer functioning This is now very rare but it can happen When you see this icon clicking on it will reset and restart the processors 4 2 THE LAYER CONTROL PANE new 3 new subsketch layer Located along the left of each sketch is a series of buttons for controlling the display of the various layers of the active subsketch Each layer is associated with one of the large buttons in the layer control pane The topmost layer button corresponds to the topmost layer on the subsketch It might be helpful to think of the layers as a stack of transparent sheets
99. s was taught You can edit and delete classes once you ve added them using the buttons in the toolbar You can also right click on a class to edit its description Double clicking a class shows the assignments and students for that class The New assignment and New student buttons are used to define assignments and students respectively Edit and Delete buttons are available on the toolbar when assignments and or students are selected all classes GEO201 Structural Geology assignment New student la Edit X Delete Class Sketches Title Due Date A Faults 3 28 2011 3 31 2011 Tools A Layers of the Earth 4 4 2011 4 7 2011 Adams Michael Bennett Josh C rragel Rebecca Od Preferences Larkin Patricia Rummler Daria smith Justin wright Dontae CogSketch version 3 07 59 7 2 ADDING SKETCHES TO THE GRADEBOOK Double clicking an assignment shows the sketches submitted for that assignment Double clicking a student shows the sketches submitted by that student all classes GEO201 Structural Geology Layers of the Earth Manage 4 Add sketches 1 X Delete Fi Grade worksheets view grade report Class Sketches Tite Student s Date Received Score layers solution solution sketch gt 4 2 2011 Tools 1f layers_Larkin Larkin Patricia 4 6 2011 63 0 100 I layers Rummler Rummler Daria 4 6 2011 95 0 100
100. sccscccescsccscsscscssces 16 APPENDIX D EXPORTED HISTORY FORMAT scsccscecsscecsccsccccsccccccessccsscccscesecs UP INDEX cere CogSketch version 3 07 59 60 61 62 1 INTRODUCTION TO COGSKETCH 1 1 WHAT IS COGSKETCH People sketch to work through ideas and to communicate especially when dealing with spatial matters Software that could participate in sketching could revolutionize spatial education and provide a new kind of instrument for cognitive science research It would also be an important scientific advance in its own right CogSketch is a sketch understanding system that we are creating to achieve these goals To be a participant in sketching CogSketch must understand what you are drawing One part of that is understanding the visual and spatial properties of the digital ink that you draw Another part of that is having a conceptual understanding of what your ink is depicting in real world terms CogSketch has built into it a powerful set of visual and spatial processing routines for encoding properties of what you draw in a human like way CogSketch also uses a large library of knowledge about the world to provide its conceptual understanding of what you are drawing It also has capabilities for analogical matching of sketches so that it can compare sketches in a human like way However it s important to remember that today s computers have far less
101. sketch in which the student does their work is always called the cetroa worksheet Workspace Its properties ate controlled ssssssgmnmunimm cC by the first two tabs of the Worksheet Workspace Workspace Concepts Solution Important Facts Property Editor as shown below Problem Statement problem statement provides an Uae cums B Please draw an example of an animal cell showing where the DNA is located explanation of the task to the student or participant in an experiment This text ts displayed on the Problem tab of the User Interface Skin 4 1 i This skin was designed for use with subsketches used asthe sidebar when a student 1S WO rking a Experiment Workspace minimal workspace in CogSketch worksheets It provides a minimal set k h Experiment Workspace of user interface elements needed for a student sketch CogSketch Default User Interaction Locks You can optionally prevent the student from performing various types of modifications to the glyphs in the workspace Workspace Items Update Layer 1 For each ofthe actions listed below ifthe checkbox is checked the student will be prevented from performing that kind of action on the selected workspace glyph basic glyph type Miscellaneous Initial Interaction Mode Y v Allow users to enter their own names for glyphs Enable the pen tool for drawing straight lines Currently we only all
102. suggestion in the Overall Sketch category Expand this category and you will see that the tutor has suggested that something isn t quite right about the location of the DNA molecule j CellDNA Workspace f x fin ish m TOT F M dia 4 glyphs We 9 d W Your Progress Update All Glyphs have Labels Euer Required Glyphs p Overall Sketch e You might reconsider the location of the DNA DNA molecule Cell nucleus Po mM a i TN Cee Cell membrane Curre nt Subsketch 3 7 1 6 in n zoom 3556 4 Your Progress Move the DNA inside the nucleus and click Update on Update the Feedback tab again The tutor has no suggestions All Glyphs have Labels yout sketch is good Required Glyphs ee Overall Sketch ae Your sketch looks good to me CogSketch version 3 07 16 4 THE COGSKETCH INTERFACE This section desctibes each of the interface elements in CogSketch 4 1 COGSKETCH TOOLBAR T finish 0 L i wr AU Q P EIE m 7 glyphs ink E y Each sketch has its own toolbar located at the top of the sketch s window 4 1 1 THE DRAWING TOOLS QA iii glyphs ink 4111 Zoom In Use this tool to zoom in on your sketch Each time you click this button the current magnificati
103. t menu You will be prompted for the password and if entered correctly you will be able to view and edit the solution subsketch CogSketch version 3 07 58 7 CoGSKETCH GRADEBOOK The gradebook is a convenient tool for organizing and grading sketches submitted by students for assignments in the various courses handled by an instructor On the File menu choose Open Gradebook to use it The gradebook can be used for multiple classes each of which can have multiple assignments The gradebook s main window looks like the following or at least it will once you ve added a few classes Gradebook Manage class Edit class X Delete class Class Sketches Title Secondary Title Start Date T GEO201 Structural Geology Spring 2011 3 21 2011 c Tools 55101 Amphibians Fall 2011 9 5 2011 E 5311 Predator Mammals Fall 2011 9 5 2011 A5210 Small Mammals Winter 2012 1 2 2012 9 Preferences 7 1 ADDING CLASSES ASSIGNMENTS AND STUDENTS TO THE GRADEBOOK The very first time you open the gradebook no classes will be defined so the initial list of classes will be empty except for some information describing how to add classes Click the New class button to add a class The primary and secondary titles assigned can be anything meaningful to you In the screenshot above the instructor decided to use the primary title for the course number and name and the secondary title for the quarter in which that clas
104. t see the Perceptual Sketchpad in Section 8 1 This means you need to draw each entity that you wish to mention separately as a glyph If you are drawing a person and how they relate to other objects for instance drawing the person as a single glyph might be sufficient But if you are trying to communicate for example properties of their posture you need to draw multiple glyphs one for each body part that will be mentioned You can group these glyphs into a larger glyph representing the entire body if you so desire Notice CogSketch version 3 07 14 that grouping 1s different than merging segmenting Grouping preserves the identity of the constituent glyph whereas merging does not 3 4 WORKSHEET EXAMPLE PARTS OF A CELL CogSketch worksheets are motivated by the structure of activities and pencil and paper worksheets found in middle school science classes To see how to create your own wotksheet please see Section 6 This section illustrates the use of the cell DNA CogSketch worksheet that comes with the CogSketch distribution Select Open Worksheet from the welcome screen and open the file called 3 Cell DNA Worksheet skw This may take up to a minute on a slow machine because CogSketch is doing its visual analysis of the instructor s sketch which is hidden from view unless you are in Experimenter or Developer mode Draw a cell with the DNA outside of the nucleus making sure that you create the cell membrane the nucleus and th
105. tch tries to guess which glyphs should participate in the relation It moves the handles to those participants and displays some text summarizing the relation Bill does business with Diane in this example You can change which glyphs participate in the relation by dragging the handles and dropping them on other glyphs Diane CogSketch version 3 07 S LL Does business w Bill Glyph Type relation What is this Does business with Likes Loves Owes money to Means that some agent does some business with some other agent Bill does business with Diane 24 4 3 2 4 Glyph Properties relations general sketches All the information in the previous section about relations in worksheets applies to general sketches too Relations must be drawn as atrows The glyph type should be set to relation The difference 1s that you can select any binaty relation in the knowledge base Start typing the name of the desired concept and the selection box will show all the binaty relations starting with the letters you typed Select one of those to assign the relation to your glyph As before drag the handles to select which glyphs participate in the relation CogSketch version 3 07 What is this occursDuring occursDuring accursWhile occursDuringisa binary predicate relating an event to a time interval in which it occurs occursDuring EVENT TIMEINTERVAL
106. u items you see may vary from the screen shots in the manual based on the setting of switches in the preferences dialog See Section 0 3 1 THE WELCOME SCREEN E CoaSketcF EE aos UELLE Edit View FIRE Windows Help For Students Worksheets Open Worksheet Worksheet Basic Tutorial Open a CagsSketch Worksheet file For Students Engineering Design X Open Design Sketch New Design Sketch Open an engineering design sketch Create a new engineering design sketch Design Coach Basic Tutorial Design Coach Advanced Tutorial Learn to explain rotation For Instructors Gradebook Makes it easy to organize and grade sketches submitted by students For Experts Create New Worksheet CogSketch Expert Tutorial Create a new worksheet for use in classes When you start CogSketch any time there is no sketch open that s not minimized the welcome screen will be visible It provides easy access to common actions and makes it easy to get to the tutorials that are most useful for new usets CogSketch version 3 07 10 3 2 TUTORIALS CogSketch comes with several tutorials available through the welcome screen and the Help menu which provide step by step interactive instructions The Help menu lists all the provided tutorials while the welcome screen shows only those thought to be most helpful for new users If you re new to CogSketch start with the Worksheet Basic Tutorial
107. uman visual spatial reasoning CogSketch has been used in a variety of cognitive simulation experiments 17 22 You can use CogSketch to provide an interface for sketching stimuli extracting the data either via saved files or the socket based KQML interface lm a cognitive scientist interested in gathering sketching data in my lab classroom experiments CogSketch gathers detailed timing data both at the level of individual ink strokes and at the level of glyphs This greatly simplifies analyzing the order in which elements of a sketch are drawn 14 25 It can provide timing signals to support synchronizing it with other equipment such as eye trackers The grading rubrics in Sketch Worksheets can be used to help automatically score data an educator interested in using intelligent sketch based educational software in my classroom CogSketch is designed as a platform for intelligent sketch based educational software We are exploring two models currently both of which have been used in classroom experiments a Sketch Worksheets 30 are aimed at helping students learn about spatial layouts and configurations They are like pencil and paper worksheets except that they can provide immediate feedback Moreover grading rubrics can be specified and a gradebook is provided to simplify using sketch worksheets with your classes You can either use pre existing sketch worksheets we are planning to set up a web based Worksheet Exchange to fac
108. ving Raven s Progressive Matrices Proceedings of the 29th Annual Conference of the Cognitive Science Society Nashville TN Lovett A Tomai E Forbus K and Usher J 2009 Solving geometric analogy problems through two stage analogical mapping Cognitive Science 33 7 1192 1231 Lovett A amp Forbus K 2011 Organizing and representing space for visual problem solving Proceedings of QR 201 1 Lovett A amp Forbus K 2012 Modeling multiple strategies for solving geometric analogy problems Proceedings of the Thirty Fourth Annual Conference of the Cognitive Science Society Sapporo Japan Lovett A 2012 Spatial Routines for Sketches A Framework for Modeling Spatial Problem Solving Doctoral dissertation Northwestern University Department of Electrical Engineering and Computer Science Evanston Illinois Rasch R Kott A amp Forbus 2002 AI on the Battlefield An experimental exploration Proceedings of the 14th Innovative Applications of Artificial Intelligence Conference Edmonton Canada Shipley T Glazek K amp Forbus K 2010 Using CogSketch to study and teach spatial thinking Presentation at 6 International Conference on Geographic Information Science Zurich Switzerland September CogSketch version 3 07 82 20 21 28 29 30 31 Wetzel J and Forbus K 2008 Integrating Open Domain Sketch Understanding with Qualitative Two Dimensional Rigid Body
109. w Knowledge About Selected Items Displays the knowledge known about the items currently selected The knowledge is shown using the syntax of the underlying knowledge base 4 7 3 10 Show Knowledge About Sketch Displays all of the logical statements constituting what CogSketch knows about the current sketch 4 7 3 11 Browse KB Allows you to browse the underlying KB the vast majority of which are derived from OpenCyc www opencyc com CogSketch version 3 07 44 4 7 4 GLYPHS MENU 4 7 41 Finish Glyph This menu item serves the same segmentation function as the Finish Glyph button on the toolbar 4 7 4 2 Import Picture as Glyph You can incorporate image files into your glyphs CogSketch can handle most of the common image formats PNG JPG BMP GIF etc 4 7 5 SPATIAL MENU Spatial reasoning tools are found here See section 5 1 for more information 4 7 6 ANALOGY MENU Analogical reasoning tools are found here See section 5 2 for more information 4 7 7 EXPERIMENTER MENU Contains tools useful to researchers analyzing experiments 4 7 7 1 Show Glyph Ordering Labels each glyph in the current subsketch with large numbers indicating the order in which they were drawn 4 7 7 2 Show Ink Stroke Ordering Labels each individual ink stroke in the current subsketch with numbers indicating the order in which they were drawn This also color codes each ink stroke shows a large circle at the start of the stroke
110. yer Genre Of the following which best describes the new layer How should this layer be viewed looking from top Cancel i You cannot change a layer s type once it has been created but you can change its name The dialog also presents two other properties not seen when creating a layer the Genre and Pose The genre of a layer describes what kind of drawing it is A drawing showing the internals of a mechanical construct is quite different than an abstract node link diagram for example The current choices for genre are e Abstract view No assumption is made about how spatial relationships between glyphs might be signaling conceptual relationships between the entities that they depict Discrete graph view Contact relationships between glyphs should be considered significant but distances and spatial layout are not e Geospatial view The entities in the sketch are located in a large scale physical space where a geographical coordinate system is assumed i e the sketch is a kind of map Physical view The spatial relationships between glyphs that depict physical entities are considered significant e g two glyphs which visually overlap may be signifying something about the relationship between the entities that they depict If you pick Geospatial View or Physical View for the genre you will have several choices for the layer s Pose The pose is the orientation of the drawing with respect to the viewers CogSk

CogSketch Manual - Qualitative reasoning group

Contents

Download Pdf Manuals

Related Search

Related Contents