Home

VeilFS Documentation

1. 66 Chapier 3 Files VeilFS Documentation Release 1 0 3 13 fmemopen h Functions FILE fmemopen void buf size_t size const char mode A BSD port of the fmemopen Linux method using funopen man docs for fmemopen man docs for funopen This method is ported from ingenuitas python tesseract project You must call fclose on the returned file pointer or memory will be leaked param buf The data that will be used to back the FILE methods Must be at least c size bytes param size The size of the c buf data param mode The permitted stream operation modes returns A pointer that can be used in the fread fwrite fseek fclose family of metho If a failure occurred NULL will be returned 3 14 globus stdio ui c Functions int globus I stdio ui activate void int globus I stdio ui deactivate void int globus I stdio ui read UI ui UL STRING uis int globus I stdio ui write UI ui Ul STRING uis Variables globus module descriptor t globus i stdio ui module const Ul METHOD globus I stdio ui old method UI METHOD globus I stdio ui method 3 13 fmemopen h 67 VeilFS Documentation Release 1 0 3 15 globus stdio ui h Defines GLOBUS STDIO UI MODULE Variables globus module descriptor t globus i stdio ui module 3 16 grid proxy info c Defines SHORT USAGE FORMAT args show short help0 args show full usage args error message errmsg args err
2. application json Status Codes 2200 OK 404 Not Found 500 Internal Server Error Example request GET rest latest attrs dirl dir2 HTTP 1 1 Header content type application json Host example com Example responses HTTP 1 1 200 OK connection close server Cowboy date Sun 05 Jan 2014 17 17 39 GMT content length 157 Access Control Allow Origin content type application json mode 8 uid 20000 gid 20000 atime 1388937272 mtime 9 4 5 rest latest shares Methods GET rest latest shares Retrieve shared files as a list of globally unique identifiers Response Headers Content Type application json Status Codes 200 OK 404 Not Found 500 Internal Server Error Example request GET rest latest files shares HTTP 1 1 Header content type application json Host example com Example responses 1388937283 ct ime 13889372 100 Chapter 9 REST API VeilFS Documentation Release 1 0 HTTP 1 1 200 OK connection close server Cowboy date Sun 05 Jan 2014 17 47 00 GMT content length 36 Access Control Allow Origin content type application json O4ef3c62eaQcdba9cd2acla860835efe POST rest latest shares Share existing file This request adds specified file to a list of shared files Parameters path string path to file to be shared Request Headers Content Type application json Response Headers Content Type
3. dn DN string rdnSequence 4 AttributeTy peAndValue Result ok I ferror any noindex Removes user from DB by login shortname to oid code string gt term no return Converts OpenSSL short name to erlang like OID code sign in Proplist Result e Proplist list e Result string user_doc This function should be called after a user has logged in via OpenID It looks the user up in database by login If he is not there it creates a proper document If the user already exists synchronization is made between document in the database and info received from OpenID provider update dn list User NewDnList Result e NewDnList string e Result ok user_doc I error anyO e User user_doc Update ftveil document encapsulating user record with new DN list and save it to DB update email list User NewEmailList Result e NewEmailList string e Result ok user_doc I error anyQ e User user_doc Update veil_document encapsulating user record with new e mail list and save it to DB update_teams User NewTeams Result e NewTeams string e Result ok user_doc I error anyQ e User user_doc Update ftveil document encapsulating user record with new teams and save it to DB 11 4 10 Dns Documents 11 4 Veil modules VeilFS Documentation Release 1 0 dns_ranch_tcp_handler Authors Bartosz Polnik Copyright This s
4. Config isSet bool getBool std string opt Returns boolean value of requested option Before using this function you should check is option exists but it s not required See Config isSet Warning If given opition wasn t set you ll get empty object of given type T TO bool isSet std string opt Checks if given option is set See Config getValue void setGlobalConfigFile std string path Sets path to global config file See Config parseConfig void setUserConfigFile std string path 2 2 veil client Config 13 VeilFS Documentation Release 1 0 Sets path to user config file See Config parseConfig void setEnv Saves current CWD and HOME env viariables This is required as FUSE changes them after non debug start This is also done automati cally in Config Config bool parseConfig Parses config from files set by Config setGlobalConfigFile and Con fig setUserConfigFile User config overides global settings If user config declares all required options global config file isn t required otherwise it has exists Config Config Public Static Functions std string absPathRelToC WD std string path Converts relative path to absolute using CWD env as base prefix std string absPathRelToHOME std string path Converts relative path to absolute using HOME env as base prefix void putEnv std string name std string value Saves given env v
5. Function Details form Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains element definition for simplest HTML form Function Index reflect 0 render_element l 144 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Function Details veil_upload Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains element definition for advanced file upload form Function Index event l reflect 0 render_element l Function Details 11 4 9 Dao Documents dao Authors Rafal Slota Copyright This software is released under the MIT License Descritpion This module implements worker plugin behaviour callbacks and con tains utility API methods DAO API functions are implemented in DAO sub modules like dao cluster dao vfs All DAO API functions Should not be used directly use dao handle 2 instead Module atom is module suffix prefix is dao MethodName atom is the method name and ListOfArgs term is list of argument for the method If you want to call utility methods from this module use Module utils See dao handle 2 for more details Behaviours worker plugin behaviour Function Index e cleanup 0 doc to term l e get_record l e handle 2 init l init storage 0 11 4 Veil modules 145 VeilFS Documentation Release 1 0 list_records 2
6. Vsn term gen_server code_change 3 create simple cache Name atom Result Result ok error during cache registration Creates simple cache create simple cache Name atom CacheLoop ClearFun term Result e CacheLoop integer atom Result ok error_during_cache_registration loop_time_not_a_number_error Creates simple cache create_simple cache Name atom CacheLoop ClearFun term Strong CacheConnection boolean Result e CacheLoop integer atom Result ok error_during_cache_registration loop_time_not_a_number_error Creates simple cache create_simple cache Name atom CacheLoop ClearFun term Strong CacheConnection boolean Pid pid Result e CacheLoop integer atom Result ok error_during_cache_registration loop_time_not_a_number_error Creates simple cache generate sub proc list Name atom MaxDepth integer MaxWidth integer ProcFun term MapFun term 1 gt Result e Result list Generates the list that describes sub procs generate sub proc list Name atom MaxDepth integer MaxWidth integer ProcFun term MapFun term gt Result e Result list Generates the list that describes sub procs handle call Request term From pid Tag term State term gt Result NewState term Reason term 114 Chap
7. Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module is able to do additional translation of record decoded using protocol buffer e g it can change record atom to Erlang atom type Function Index translate 2 translate_to_record l Function Details translate Record tuple DecoderName string Result e Result term Translates record to simpler terms if possible translate to record Value term Result e Result tuple term Translates term to record if possible request_dispatcher Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module forwards client s requests to appropriate worker_hosts Behaviours gen_server http www erlang org doc man gen_server html 11 3 Cluster elements 119 VeilFS Documentation Release 1 0 Function Index code_change 3 handle_call 3 handle_cast 2 handle_info 2 init l send_to_fuse 3 start_link 0 stop 0 terminate 2 Function Details code change OldVsn State term Extra term Result e OldVsn Vsn down Vsn e Result ok NewState term I error Reason term Vsn term gen server code change 3 handle call Request term From pid Tag term State term gt Result NewState term Reason term Reply term Result reply Reply NewState reply Reply NewSt
8. Files Source code 3 1 gsiHandler cc Author Rafal Slota Copyright C 2013 ACK CYFRONET AGH This software is released under the MIT Licence Defines PROXY_INIT globus proxy utils program names PROXY_INFO X509 USER CERT ENV X509 USER KEY ENV GLOBUS P12 PATH GLOBUS PEM CERT PATH GLOBUS PEM KEY PATH MSG DEBUG INFO CRYPTO FREE M X 37 VeilFS Documentation Release 1 0 STDOUT_MAX_LEN namespace veil namespace client namespace gsi Functions string findUserCert string findUserKey 3 2 jobScheduler cc Author Rafal Slota Copyright C 2013 ACK CYFRONET AGH This software is released under the MIT Licence Defines PTHREAD_CMD M Run command M fetch error code and if error occured restart daemon namespace veil namespace client 3 3 messageBuilder cc Author Beata Skiba Copyright C 2013 ACK CYFRONET AGH This software is released under the MIT Licence Functions string tolower string input 38 Chapter 3 Files VeilFS Documentation Release 1 0 namespace veil namespace client 3 4 veilfs cc Author Rafal Slota Copyright C 2013 ACK CYFRONET AGH This software is released under the MIT Licence Defines SH_RUN NAME ARGS FUN Runs FUN on NAME storage helper with constructed with ARGS Return value is avaiable in int sh_return RETURN_IF_ERROR X If given veilError does not produce POSIX 0 return code interrupt execution b
9. Gets connection_info record with given SessID form DB Should not be used directly use dao handle 2 instead get_fuse session Fuseld uuid stale update before ok gt fok veil_document no_return 148 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Gets fuse session record with given FuseID form DB Second argument shall be either stale update before will update cache before getting value or stale ok which is default get value from cache Default behaviour can be also achieved by ommiting second argument Should not be used directly use dao handle 2 instead get state gt ok term I ferror any Retrieves cluster state with ID cluster state from DB Should not be used directly use dao handle 2 instead get state Id atom ok term I error anyQ Retrieves cluster state with UUID Id from DB Should not be used directly use dao handle 2 instead list connection info by session id SessID uuid lok veil_document I error any Lists connection info records using given select condition Current implementeation supports fallowing selects by session id SessID select all connections that belongs to session with ID SessID Should not be used directly use dao handle 2 instead list_fuse sessions by_valid_to Time non_neg_integer fok veil_document I error any Lists fuse session records using given sel
10. Inserts doc to db insert_docs DbName string Doc doc ok term error term Inserts list of docs to db insert_docs DbName string Doc doc Opts Option ok termQ I error term e Option atom atom term0 Inserts list of docs to db 11 4 Veil modules 151 VeilFS Documentation Release 1 0 list dbs gt ok stringO I error termQ Lists all dbs list dbs Prefix string ok stringO I error term Lists all dbs that starts with Prefix name Name string atom gt binary Converts string atom to binary open design doc DbName string DesignName string ok doc ferror not found missing I deleted error term Returns design document with a given design doc name open doc DbName string DocID string gt ok doc error not found missing I deleted error term Returns document with a given DocID open doc DbName string DocID string Opts Option ok doc I ferror not found missing deleted I error term0 e Option atom atom term Returns document with a given DocID query view DbName string DesignName string ViewName string 3 ok QueryResult term I error term0 Execute a given view with default set of arguments Check record view query args for de tails query view DbName string DesignNa
11. binary0 reqQ Should return list of provided content types with specified ID e g rest resource some_id Should take into account different types of methods PUT GET etc if needed Should return empty list if method is not supported Id is a dir gt application json Id is a regular file gt Id does not exist gt delete req binary binary boolean reqQ Will be called for DELETE request on given ID Should try to remove specified resource and return true false indicating the result Should always return false if the method is not supported exists req binary binary boolean reqQ Should return whether resource specified by given ID exists Will be called for GET PUT and DELETE when ID is contained in the URL get req binary term stream integer function halt reqO regO Will be called for GET request without specified ID e g rest resource Should return one of the following 1 ResponseBody of the same type as content_types_provided 1 re turned for this request 2 Cowboy type stream function serving content of the same type as content types provided I returned for this request 3 halt atom if method is not supported get req binary binary gt term I stream integer function halt regO req Will be called for GET request with specified ID e g rest resource some_id Should return one of th
12. fd_select gt ok error any no_return Removes file descriptor from DB Argument should be uuid of file_descriptor or same as in dao_vfs list_descriptors 3 Should not be used directly use dao handle 2 instead See dao handle 2 for more details remove file File file gt ok ferror any no_return Removes file from DB Argument should be file see dao_types hrl for more details Should not be used directly use dao handle 2 instead See dao handle 2 for more details remove file meta FMeta uuid gt ok error any no_return Removes file_meta from DB Argument should be uuid of veil_document see dao_types hrl for more details Should not be used directly use dao handle 2 instead See dao handle 2 for more details 11 4 Veil modules 159 VeilFS Documentation Release 1 0 remove storage fuuid DocUUID uuid fid StoragelD integer gt ok terror any no_return Removes storage info from DB Argument should be uuid of storage document or ID of storage Should not be used directly use dao handle 2 instead See dao handle 2 for more details rename file File file NewName string gt fok NewUUID uuid no_return Renames specified file to NewName Should not be used directly use dao handle 2 instead See dao handle 2 for more details save _descriptor Fd fd_info fd_doc gt fok uuid ferror any n
13. rest_init req term ok reqQ term shutdown req Cowboy callback function Called right after protocol upgrade to init the request context Will shut down the connection if the peer doesn t provide a valid proxy certificate veil_multipart_bridge Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This is a custom multipart bridge It checks if a request is a multipart POST requests checks its validity and passes control to file_transfer_handler Function Index e parse l Function Details parse ReqBridge record ok not multipart ok Params tupleQ Files uploaded_file I ferror any0 Try to parse the request encapsulated in request bridge if it is a multipart POST request 132 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Rest modules Documents rest_attrs Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This module implements rest_module_behaviour and handles all REST requests directed at rest attrs path It returns file attributes if possible Behaviours rest module behaviour Function Index allowed methods 3 content types provided 2 content types provided 3 delete 3 exists 3 get 2 get 3 post 4 put 4 validate 4 Function Details allowed methods req binary binary gt binaryQ req0 Should return list of methods
14. term Result Result ok term worker_plugin_behaviour callback init 1 Sets up the worker for propagating logs to CMT ses sions and configures lager trace files central_logging_backend Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This is a gen_event module lager backend responsible for intercept ing logs and sending them to central sink via request_dispatcher Behaviours gen_event http www erlang org doc man gen_event html Function Index code_change 3 handle_call 2 handle_event 2 handle_info 2 init l 126 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 terminate 2 Function Details code change OldVsn State term Extra term Result e OldVsn Vsn down Vsn e Result ok NewState term I ferror Reason term Vsn term gen event code change 3 handle call Request term State term Result e NewState term e Reply term Result ok Reply NewState gen event handle call 2 handle event Request term State term Result NewState term Result ok NewState gen_event handle_event 2 handle info Info term State term Result NewState term Result ok NewState gen event handle info 2 init Args term Result Result ok term gen event callback init 1 Called after installing this handler into lager event l
15. uuid gt ferror any no return Removes info about file sharing from DB by share id file name or user uuid Should not be used directly use dao handle 2 instead See dao handle 2 for more details save file share Share file share info file share doc gt fok file share ferror any no return Saves info about file sharing to DB Argument should be either share_desc record if you want to save it as new document or veil_document that wraps share_desc if you want to update descriptor in DB See dao save_record l and dao get_record I for more details about ftveil document wrapper Should not be used directly use dao handle 2 instead See dao handle 2 for more details dao_users Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This module provides high level DB API for handling user documents 156 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Function Index get_files_number 2 get_user l remove_user l save_user l Function Details get user Key login Login string email Email string fuuid UUID uuid dn DN string gt fok user_doc ferror any no_return Gets user from DB by login e mail uuid or dn Non error return value is always ok veil_document record user See dao save_record l and dao get_record l for more de tails about veil_document wrappe
16. 72 Chapter 3 Files CHAPTER 4 Namespaces Documents 4 1 veil client gsi bool debug bool validateProxyConfig bool validateProxyCert std string getProxyCertPath std string getClusterHostname string findUserCert string findUserKey 73 VeilFS Documentation Release 1 0 4 2 veil client LockType enum See AutoLock AutoLock Values e READ LOCK e WRITE LOCK typedef boost function lt bool const protocol communication_protocol Answer amp gt lis tener_fun typedef struct veil client locationInfo locationInfo Structure containing file mapping base information typedef struct veil client storagelnfo storageInfo 4 3 veil client utils template lt typename T gt std string toString T in template lt typename T gt T fromString std string in 74 Chapter 4 Namespaces CHAPTER 5 Structs Documents 5 1 fmem struct fmem Public Members size_t pos size_t size char buffer 5 2 veil client Job struct veil client Job include lt include jobScheduler h gt The Job struct Used by JobScheduler to describe scheduled tasks Public Functions Job time_t when boost shared_ptr lt ISchedulable gt subject ISchedulable TaskID task std string arg0 std string argl std string arg2 bool operator lt const Job amp other Compare operator for priority queue It compares tasks by it
17. DocOrObj doc I json object Returns field s value from given document or JSON object get fields DocOrObj gt FieldName stringQ FieldValue any I error not found error invalid object DocOrObj Hdoc I json object Returns field s values from given document or JSON object mk bin Term term gt binary Converts given term to binary form used by BigCouch CouchDB mk doc Id string gt doc Returns new BigCouch document with given Id mk field DocOrObj Name string Value term gt DocOrObj DocOrObj doc I json object Inserts new field into given document or JSON object mk fields Docorobj Names string Values term D DocOrObj DocOrObj doc I json object Inserts new fields into given document or JSON object mk obj gt Returns empty json object structure used by BigCouch CouchDB mk str Str string atom gt binary Converts given string to binary form used by BigCouch CouchDB 154 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 reverse fields DocOrObj DocOrObj DocOrObj doc I json object Reverses fields in given document or JSON object rm field DocOrObj Name string DocOrObj DocOrObj doc I json object Removes field from given document or JSON object rm fields DocOrobj Name string 1 DocOrObj DocOrObj doc I json object Removes f
18. error NIF not loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode re turn value shall be 0 if operation was succesfull otherwise negated POSIX error code will be returned mkdir creates directory with given path and mode permissions mknod _sh_name string sh args string path string mode integer _rdev integer ErrorCode integer I error NIF not loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode return value shall be 0 if operation was succesfull otherwise negated POSIX error code will be re turned mknod 5 can and shall be used in order to create file not directory mode and _rdev arguments are the same as in mknod syscall open sh name string _sh_args string path string fi st_fuse_file_info ErrorCode integer FFI st_fuse_file_info error NIF not loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode return value shall be 0 if operation was succesfull otherwise negated POSIX error code will be re turned open 4 opens file fi aregument is an ist fuse file i
19. reqQ Should return true false depending on whether the request is valid in terms of the handling module Will be called before POST or PUT should discard unprocessable requests rest_files Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This module implements rest_module_behaviour and handles all REST requests directed at rest files path Essentially it serves user content files via HTTP Behaviours rest module behaviour Function Index allowed methods 3 content types provided 2 content types provided 3 delete 3 exists 3 get 2 get 3 134 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 handle_multipart_data 4 post 4 put 4 validate 4 Function Details allowed methods req binary binary gt binaryQ reqO Should return list of methods that are allowed and directed at specific Id e g if Id undefined gt lt gt lt gt if Id undefined gt lt gt lt gt lt gt content types provided req binary binaryQ reqO Should return list of provided content types without specified ID e g rest resource Should take into account different types of methods PUT GET etc if needed Should return empty list if method is not supported If there is no id only dirs can be listed gt applica tion json content types provided req binary binary
20. s Job when field 75 VeilFS Documentation Release 1 0 bool operator const Job amp other Compares equal all fields excpet Job when Public Members time_t when Time when Job should be processed boost shared_ptr lt Schedulable gt subject Pointer to object being context of Job execution ISchedulable TaskID task ID of task See ISchedulable TaskID std string arg0 Task s first argument std string argl Task s second argument std string arg2 Task s third argument Default constructor 5 3 veil client locationInfo struct veil client IocationInfo Hinclude lt include storageMapper h gt Structure containing file mapping base information Public Functions bool isValid Public Members int storageld Storage identificator See StorageMapper m_storageMapping std string fileld File identificator This ID should be recognisable by storage helper Most commonly it s just file path relative to the storage 76 Chapter 5 Structs VeilFS Documentation Release 1 0 See StorageMapper m_fileMapping time_t valid To Mapping expiration time int opened How many files are currently opened using this mapping 5 4 veil client storagelnfo struct Veil client storageInfo include lt include storageMapper h gt Public Functions bool is Valid Public Members time t last updated Last update time std string storageHelper Name Name of storage helper See StorageHe
21. utils then dao module will be used E g calling dao handle _ vfs some method some arg will call dao vfs some method some arg but calling dao handle _ utils some method some arg will call dao some method some arg You can omit Module atom in order to use default module which is dao cluster E g calling dao handle _ some method some arg will call dao cluster some method some arg Additionally all exceptions from called API method will be caught and converted into ferror Exception tuple E g calling handle _ save record Id Rec will execute dao cluster save record Id Rec and normalize return value init Args term Result 146 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Error term Result ok ferror Error worker plugin behaviour callback init 1 init storage ok error Error term Inserts storage defined during worker instalation to database if db already has defined storage the function only replaces StorageConfigFile with that definition list records ViewInfo view_info QueryArgs Hview query argsf ok QueryResult view result I ferror term Executes view query and parses returned result into view_result record Strings from fHview query args are not transformed by dao helper name l the caller has to do it by himself load view def Name string Type map reduce gt string Loads view definition from
22. 141 VeilFS Documentation Release 1 0 page_events Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page file manager Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page index Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page login Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page logout Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page logs Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page manage account Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code 142 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 page rules composer Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page rules simulator Authors Lukasz Opiola Copyright This software is released under
23. ENABLE_ENV_OPTION_OVERRIDE FUSE_ID_OPT CLUSTER_PING_INTERVAL_OPT JOBSCHEDULER_THREADS_OPT ALIVE_META_CONNECTIONS_COUNT_OPT ALIVE_DATA_CONNECTIONS_COUNT_OPT ENABLE DIR PREFETCH OPT ENABLE PARALLEL GETATTR OPT 3 6 config h 43 VeilFS Documentation Release 1 0 FUSE OPT PREFIX Prefix for all env variables that will be send to cluster PROTOCOL VERSION ATTR DEFAULT EXPIRATION TIME DECLARE DEFAULT KEY VALUE namespace veil namespace client class Config The Config Parses config files and provides safe access to configuration map Public Functions std string getFuseID Returns current FuselD void negotiateFuseID time_t delay 0 Starts FuseID negotiation process Parameters delay Since this is async actions you can specify execution delay in seconds std string getString std string opt Returns string value of requested option Before using this function you should check is option exists but it s not required See Config isSet int getInt std string opt Returns int value of requested option Before using this function you should check is option exists but it s not required 44 Chapter 3 Files VeilFS Documentation Release 1 0 See Config isSet double getDouble std string opt Returns double value of requested option Before using this function you should check is option exists but it s not required See Config isSet bool ge
24. Ge pe a Ge der a an ee Y 38 3 3 MESSAPeBUNdence sak H sagen se AGE AGR TE A Ba fed GLE je a Ge 38 JA VelliSicC es cae s r ed be SAG bed frk eka eek ee oe kA Ed ee 39 35 VONEUSO CC asbie SATS SER A EE SEE Eee 39 A O amp A Ae ee a SES TNS STE ASS ANS GE ERE 43 Bel De 13 0 20 00 Ms ti a A A e KO Rie eG 48 30 ssH ndl rh szos o ii a ee Boe ia tdo lana de ds 53 39 lock Mesi iii ra e A A a a ds de k 54 3 10 messageBUllder s sos s sek se A a a a et 56 UI II 58 312 Imemopene s saa Aa ad A SAGE a Bae eG bad ALA ee Ae Ge 65 3 13 fmemopenh iio be eae ee eda a wae de rd Ee ne See A 67 3 14 SI0bus StAO ME 5 p HR ee se Re ec ew Ge JAR e eg AY Bk f re de iris Ge GO ind 67 II 10 11 3 15 elo bus stdio MB hi e a AG AE AG ge AEE Ge 68 3 16 gd Proxy INIO oo be AG Eek ar RELA ee bee wae rd e ES 68 3 17 P PEO Y AMO ske Gp sp SR GO A l n ig gr kN Ge i endr date 69 318 sos e Saera eee wow a A eaten a AA A ee 70 Namespaces 73 Adell yek eent ESE 2454 234 4 AAS GAGA be dd db od AS 73 42 FERCHED ss s sug a te Hae Er S klade a GAGE Er sd a see 74 43 vellchent utlls osea a A are ske KAGGE A 74 Structs 75 MN MEM a de A ed A A BS 75 2 Vell iclentlOD occ e e e E E ARE Re E A ee gs Be fed 75 S3 Wellclientelocationinlo ezme os EEE e E A A aa F es 76 SA veilk drentsst r selnfo so ia ra a dd e a 77 VeilCluster 79 About 83 6I Goal se ck eG AA ee ee Ee eR Ew AO ee eS ee
25. Result e Options term Request term Response term Result ok Response Options Handles a request producing a response with use of Nitrogen engine or a file stream response init Protocol Request term Options term Result 130 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 e Protocol Transport term0 http e Result ok Request term state Initializes a request response procedure terminate Reason Request State Result Reason term Request term Result ok State term Cowboy handler callback rest_handler Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This is a cowboy handler module implementing cowboy rest interface It handles REST requests by routing them to proper rest module Function Index allowed methods 2 content types accepted 2 content types provided 2 delete resource 2 get resource 2 handle json data 2 handle multipart data 2 handle urlencoded data 2 init 3 resource exists 2 rest init 2 Function Details allowed methods req state binaryQ req0 state Cowboy callback function Returns methods that are allowed for request URL Will call al lowed_methods 2 from rest_module_behaviour content types accepted req state term req0 state Cowboy callback function Returns content types that are accepted by R
26. callback devel fuse libs fuse devel cmake28 boost In s usr lib64 globus include globus config h usr include globus 83 VeilFS Documentation Release 1 0 6 2 2 VeilCluster RPM installation and configuration After installing prerequisites you can install the VeilCluster RPM rpm i veil lt version gt rpm VeilCluster setup scripts can be executed with the following command veil setup The script will guide you through the setup procedure of the VeilCluster To start the work you should start at least one instance of the CCM component and one worker component instance A sample session of starting VeilCluster from scratch is as follows root plgsl63 veilcluster veil setup xxx Veil SETUP xxx Nodes configured on this machine will use its hostname 172 16 67 219 1 Make sure it is resolvable by other hosts in the network gt What do you want to do 1 Manage database nodes 2 Manage veil nodes 3 Exit gt Your choice 2 Each machine can only host a single worker or a ccm worker pair gt What do you want to do 1 Set up a new cluster 2 Extend existing cluster 3 Go back gt Your choice 1 Installing a new cluster beside a running one may cause unpredictable behaviour It is required that all database nodes are installed prior to the cluster gt Do you wish to continue 1 Yes 2 No gt Your choice 1 gt List ALL running database nodes delimiting them with
27. commas no spaces eg dbl host net c gt The cluster will use ONLY the nodes specified now gt Running DB nodes db 172 16 67 219 gt Connection to following database nodes has been confirmed db 172 16 67 219 gt Following nodes will be installed ccmt172 16 67 219 worker 172 16 67 219 gt Confirm 1 Continue 2 Go back gt Your choice 1 Installing ccme172 16 67 219 Installing worker 172 16 67 219 Starting node s 2 2 To check if the VeilCluster is running you can attach to the CCM component and check heartbeat mes sages root plgs163 veilcluster opt veil nodes ccm bin veil_cluster_node attach Attaching to tmp opt veil nodes ccm erlang pipe 1 D to exit search Ye 84 Chapter 6 About VeilFS Documentation Release 1 0 ccme172 16 67 219 1 gt 11 07 00 454 info Heart beat on node ccme172 16 67 219 sent connectic 11 07 00 454 info Heart beat on node ccm 172 16 67 219 answered new state_num 4 ccme172 16 67 219 1 gt 11 07 09 989 info Cluster state ok After starting the nodes you have a complete VeilCluster running You can now proceed to VeilClient installation Note To have a fully working VeilFS installation we should also start a BigCouch instance on the same machine with its cookie set to veil_cluster_node and hostname set to db 6 3 Support For more information visit project Confluence or write to wrzeszcz ag
28. const char path const char link symlink FUSE callback Not implemented yet See int rename const char path const char newpath rename FUSE callback See int link const char path const char newpath link FUSE callback Not implemented yet See int chmod const char path mode t mode chmod FUSE callback See int chown const char path uid_t uid gid t gid 60 Chapter 3 Files VeilFS Documentation Release 1 0 chown FUSE callback Not implemented yet See int truncate const char path off_t newSize truncate FUSE callback See int utime const char path struct utimbuf ubuf utime FUSE callback Not implemented yet See int open const char path struct fuse file info filelnfo open FUSE callback See int read const char path char buf size_t size off t offset struct fuse file info filelnfo read FUSE callback See int write const char path const char buf size_t size off t offset struct fuse file info filelnfo write FUSE callback See int statfs const char path struct statvfs statInfo statfs FUSE callback Not implemented yet See int flush const char path struct fuse file info filelnfo 3 11 veilfs h 61 VeilFS Documentation Release 1 0 flush FUSE callback Not implemented yet See int release const char path struct fuse file info filelnfo release FUSE callback
29. e get state 0 e get state l e list connection inforl list fuse sessions I e remove connection inforl remove fuse session I e save connection inforl save fuse session I save state l save state 2 Function Details clear sessions gt ok no_return Cleanups old unused sessions from DB Each session which is expired is checked If there is at least one active connection for the session its expire time will be extended Otherwise it will be deleted Should not be used directly use dao handle 2 instead clear_state gt ok no_return Removes cluster state with Id cluster_state Should not be used directly use dao handle 2 instead clear state Jd atom gt ok no_return Removes cluster state with given Id Should not be used directly use dao handle 2 instead close connection SessID uuid gt ok no_return Removes connection_info record with given SessID form DB and tries to close it This method should not be used unless connection exists Otherwise it will fail with exception error Should not be used directly use dao handle 2 instead close_fuse_session Fuseld uuid gt ok no_return Removes fuse_session record with given FuseID form DB and cache Also deletes all con nections that belongs to this session and tries to close them Should not be used directly use dao handle 2 instead get connection info SessID uuid gt ok veil_document no_return
30. ee ee 83 62 Getting Started vass cutres eH E Mee OH ERO Pw A ee See ARS 83 PD SUPPE gt he pre fe See s ee ere le de 85 Development 87 dl Repo layouts sr stage a bader ke STAS nee ee Sew de eee Se Ka 87 T DIG sa KG Skee OO SE ste A G KG Saar Go AEE ae fee Ge G 87 TD TS ben bee bbe ea wed BPE bn Oe bE EGA et Maa eee eee ee q 88 WA RUG Luts dr oe eed eSATA A eS 88 7 5 Development using Makefile to generate single releases and test environments of veil cluster nodes 88 Er Rel ise management sreang ei Pa ote Ee ee does Kip gs ged he see 91 TT SUPPORT A eee re Pee 92 Installation 93 8 1 Dependencies asc Go BE ke ar dere A Bea SE Swe a SA Oe a EGG 93 2 Systemi preparation saa r eae Sb aS SAG GATAS Sa KG GE LAG GAGE do Gow 93 8 3 RPM build and installation se sa i ses sk a Ge Ga bl Ge dd eee ant s 93 SA Usetul commands Eder 2064 24008 FET tb dde GeE NE Ge Fa 94 So Vocabulary si aeee e seere ETA NESS FRASA AS ee 94 REST API 95 gI Authentication sceso 4 5 e a hehe ea ee r ode ee ea 95 9 2 Errors amp Validation sg sos spe sa BP a Be al Get 96 9 3 Getting started s vs saks be hee A A be ER aS 96 OA RESOURCES va na ok ok ir Be ea EE SE A SES Bas eR ES G 96 95 Structure of the REST URIS scoe s cd oh ae eA ee a E 103 C modules 105 WOT GridProxy Verily lt 10 a date ka ee BER SE E AAA RAN a Rie let 105 10 2 Vel HelpersNTR Gn gt gnp Sido foe gg p iy eo ek Sad Fe pd BE dee 106 Erlang modules
31. file remove record Id atom uuid ok I error Error termQ Removes record with given UUID from DB Should not be used directly use dao handle 2 instead save record term veil_document uuid string rev info term record term force update boolean gt fok Docld string error conflict no return Saves record to DB Argument has to be either Record term which will be saved with random UUID as completely new document or veil document record If fveil document record is passed caller may set UUID and revision info in order to update this record in DB If you got veil_document via dao get record l uuid and rev info are in place and you shouldn t touch them Should not be used directly use dao handle 2 instead set db DbName string ok Sets current working database name dao cluster Authors Rafal Slota Copyright This software is released under the MIT License Descritpion This module gives high level DB API which contain veil cluster spe cific utility methods All DAO API functions should not be called directly Call dao handle _ cluster MethodName ListOfArgs instead See dao handle 2 for more details Function Index clear sessions 0 clear state 0 clear state l close connection I close fuse session I 11 4 Veil modules 147 VeilFS Documentation Release 1 0 e get connection infor l e get fuse session I e get fuse session 2
32. file file txt Example responses HTTP 1 1 100 Continue HTTP 1 1 204 No Content connection close server Cowboy date Fri 24 Jan 2014 08 43 05 GMT content length 0 Access Control Allow Origin x content type application octet stream DELETE rest latest files path Delete regular file at specified path if it exists Parameters path string path to file or directory Response Headers Content Type application json Content Type application octet stream Status Codes 204 No Content 404 Not Found Example request DELETE rest latest files dir file txt HTTP 1 1 Header content type application json Host example com Example responses HTTP 1 1 404 Not Found connection close server Cowboy date Fri 24 Jan 2014 08 50 18 GMT content length 0 Access Control Allow Origin content type application json 9 4 4 rest latest attrs path Methods GET rest latest attrs path Retrieve attributes of specified file or directory as a record of structure property value Fields of returned record efile protection mode efile owner user ID efile owner group ID efile last access time 9 4 Resources 99 VeilFS Documentation Release 1 0 efile last modification time efile or inode last change time efile type efile owner user name efile owner group name Parameters path string path to file or directory Response Headers Content Type
33. file to DB See dao save_record l and dao get_record I for more details about veil_document wrapper Should not be used directly use dao handle 2 instead See dao handle 2 for more details save storage Storage storage_info veil_document gt fok uuid fer ror any no_return Saves storage info to DB Argument should be either storage_info record if you want to save it as new document or veil_document that wraps storage_info if you want to update storage info in DB See dao save_record l and dao get_record l for more details about ftveil document wrapper Should not be used directly use dao handle 2 instead See dao handle 2 for more details unlock_file UserID string FilelD string Mode write read gt not_yet_implemented Takes off a read write lock on specified file owned by specified user Should not be used directly use dao handle 2 instead See dao handle 2 for more details Not yet implemented This is placeholder template method only 160 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 user_logic Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This module provides methods for managing users in the system Function Index create user 5 e extract dn from cert I e get dn list I e get email list I e get login I e get name l get team names I e get teams I e get_user l invert dn st
34. list size_t size listxattr FUSE callback Not implemented yet 2 12 veil client VeilFS 33 VeilFS Documentation Release 1 0 See int removexattr const char path const char name removexattr FUSE callback Not implemented yet See int opendir const char path struct fuse_file_info filelnfo opendir FUSE callback Not implemented yet See int readdir const char path void buf fuse_fill_dir_t filler off_t offset struct fuse file info filelnfo readdir FUSE callback See int releasedir const char path struct fuse file info filelnfo releasedir FUSE callback Not implemented yet See int fsynedir const char path int datasync struct fuse file info filelnfo fsyncdir FUSE callback Not implemented yet See int init struct fuse_conn_info conn init FUSE callback See bool runTask TaskID taskId std string argO std string arg1 std string arg3 34 Chapter 2 Classes VeilFS Documentation Release 1 0 Task runner derived from Schedulable See ISchedulable runTask Public Static Functions boost shared_ptr lt JobScheduler gt getScheduler TaskID taskld TASK_LAST_ID Returns JobScheduler assigned to this object boost shared_ptr lt Config gt getConfig Returns Config assigned to this object boost shared_ptr lt SimpleConnectionPool gt getConnectionPool boost shared_ptr lt PushListener gt getPushListener
35. lt typename T gt T get std string opt Internal implementation of Config getValue See Config getValue 46 Chapter 3 Files VeilFS Documentation Release 1 0 template lt typename T gt T getValue std string opt Returns type specialized value of given config option bool runTask TaskID taskId std string argO std string argl std string arg3 Task runner derived from Schedulable See ISchedulable runTask void setupDefaults Protected Attributes bool defaultsLoaded ReadWriteLock m_access std string m_globalConfigPath Path to global config file See Config setGlobalConfigFile std string m_userConfigPath Path to user config file See Config setUserConfigFile YAML Node m_globalNode Global config object YAML Node m_userNode User config object YAML Node m_envNode Temp config object used to manipulate env settings YAML Node m_defaultsNode Default configs Protected Static Attributes std string m_requiredOpts 3 6 config h 47 VeilFS Documentation Release 1 0 Array containing required options names std string m envCWD Saved CWD env variable std string m env HOME Saved HOME env variable std map lt std string std string gt m_envAll All saved env variables Friends friend class I VeilFactory namespace utils Functions template lt typename T gt std string toString T in template lt typename T gt T fromString std string in 3
36. main_ccm gt main CCM name of the cluster this node operates in opt_ccms gt list of optional CCMs this parameter is not mandatory db_nodes gt list of DBMS nodes Primarily these parameters are retrieved from config args file It should contain these parameters in the following manner lt parameter_name gt lt parameter_value gt Another way is passing these parameters via command line arguments In this case the syntax is veil_cluster lt parameterl name gt lt parameterl_value gt lt parameter2_name gt lt parameter2_value gt NOTE e parameters passed via command line OVERRIDE those in config args file 7 6 Release management 91 VeilFS Documentation Release 1 0 command line way can specify any subset of parameters for instance veil_cluster will cause the script to use all parameters from config args veil_cluster name somename O host net somenameOhost net will cause the script to use parameters from config args except node name veil_cluster opt_ccms no opt_ccms value specified will override opt ccms from con fig args with null value both parameter order can be arbitrary both multiple values eg DBMS nodes are passed as a space delimited list eg db nodes dbn odel host net dbnodel host net dbnode2 O host net dbnode2 host net both parameter values can t contain spaces or hyphens command li
37. node node name gt combines the two above Note that it only works for packages in releases test_cluster those created with gen_dev or gen test 7 5 4 Generating a local test environment The script gen test simplifies setting up a bunch of cluster nodes for testing It uses both gen_dev and veil cluster scripts To generate a testing environment proper arguments must be passed to the script workers workerl host worker2 host main ccm main_ccm_node host opt_ccms opt_ccm_nodel hos The expression after workers specifies the list of workers in the cluster The expression after main_ccm specifies the node name of the main CCM The expression after opt_ccms specifies the list of optional CCMs in the cluster not mandatory The expression after db_nodes specifies the list of all DBMS nodes Again these arguments can be obtained from gen test args or passed via args to Makefile 7 5 5 Generating releases for the whole cluster make gen_test_env args workers workerl host main ccm main ccm nodefhost opt_ccms opt_ccn make gen test env from file Both of these commands produce a release for each node in corresponding directories re leases test cluster 7 5 Development using Makefile to generate single releases and test environments of veil clust 9 nodes VeilFS Documentation Release 1 0 7 5 6 Starting previously generated cluster make start test env ar
38. of file_meta record Non error re turn value is always fok veil_document record file_meta See dao save_record l and dao get_record l for more details about veil_document wrapper Should not be used di rectly use dao handle 2 instead See dao handle 2 for more details get path info File file_path gt ok file doc ferror any no_return 158 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Gets all files existing in given path from DB Argument should be file_path see dao_types hrl for more details Similar to get_file 1 but returns list containing file_doc for ev ery file within given path not only the last one Should not be used directly use dao handle 2 instead See dao handle 2 for more details get storage fuuid DocUUID uuid fid StoragelD integer gt ok stor age_doc ferror any no_return Gets storage info from DB Argument should be uuid of storage document or ID of stor age Non error return value is always ok veil_document record storage_info See dao save_record l and dao get_record I for more details about veil_document wrapper Should not be used directly use dao handle 2 instead See dao handle 2 for more details list_descriptors MatchCriteria fd_select N pos integer Offset non neg integer gt fok fd_doc ferror any no return Lists file descriptor from DB First argument is a two
39. of following number which stands for an exact API version e latest which stands for most recent stable version We strongly recommend usage of the latter option especially due to introduction of new fea tures and elimination of bugs in system Example URI that would retrieve content of dir directory https example com rest latest files dir For more information see a full list of available resources 9 5 Structure of the REST URIs 103 VeilFS Documentation Release 1 0 104 Chapter 9 REST API CHAPTER 10 C modules Documents 10 1 GridProxyVerify GridProxyVerify is an library that provides GSI proxy certificates verification based on Globus 10 1 1 Files x xxgrid _proxy _verify c This file can be compiled into independent static dynamic library Insert any peer peerCA CA CRL certificates into current verification context Proceed with verification of certificates added earlier Retrieve verification errors Basically it provides simple GSI validation that can be used through very simple API x xxgpv_nif c This file contains Erlang NIF interface for grid _proxy _verify library It ca For more info check _cluster _elements request _dispatcher gpv _nif erl_ file or just its c 10 1 2 Prerequisites x libglobus_gsi_callback libglobus common libssl Use this command to install the required dependency packages x Debian Ubuntu Dependencies deb packages ap
40. on the same machine with its cookie set to veil_cluster_node and hostname set to db 7 5 10 Useful make commands 90 Chapter 7 Development VeilFS Documentation Release 1 0 make compile compilation make test compilation execution of unit tests make generate compilation amp creation of release make docs documentation generation make PREV name of directory with previous release upgrade generation of package for hot make dialyzer f starting dialyzer in order to analyze binaries in ebin Note e in Linux you should use rebar instead of rebar rebar is built with Erlang OTP R16B if you have other Erlang version installed it may not work properly in this case download rebar sources and rebuilt rebar 7 6 Release management After generation of a release package configuration files contain default parameters The script veil cluster that comes with the package in the bin directory is used to set up and start a release 7 6 1 Prerequisites Firstly the user must have execution rights on both bin veil cluster and bin veil cluster node scripts TODO to along veil cluster czy w config Secondly config args file must be present in the config directory along with veil cluster script 7 6 2 Setting parameters List of parameters that can be set name gt name of the node erlang long name
41. python devel rpm buj 8 2 System preparation In order to make our machine recognisable for other machines in the network it is required to set its hostname properly It is recommended to set hostname to IP address so that modifica tions of etc hosts file are not necessary vi etc sysconfig network gt set HOSTNAME 172 16 67 X gt save file hostname 172 16 67 X gt instant effect system reboot is not necessary Moreover it is required to unlock ports in iptables S service iptables stop chkconfig del iptables turn off service permanently necessary if machine will be rebc 8 3 RPM build and installation After project cloning Ur make rpm create rpm in releases veil 0 0 6 1 x86_64 rpm yum localinstall releases veil 0 0 6 1 x86_64 rpm start RPM installation U veil_setup start installation script 93 VeilFS Documentation Release 1 0 Warning database needs to be installed before cluster during cluster installation it is required to give full database node name e g db 172 16 67 143 db 172 16 67 143 during cluster installation in storage definition phase give storage mount point used by cluster to store data e g defined users or groups define optional storages used by fuse client groups for each of them give mount point and name of fuse client group that is supposed to use it duri
42. remove_descriptor l remove_file remove_file_meta l remove_storage I rename_file 2 save_descriptor l save_file l save_file_meta l save_new_file 2 save_storage l unlock_file 3 Function Details count_subdirs fuuid UUID uuid gt ok non_neg_integer Returns number of first level subdirectories for specified directory find files FileCriteria file_criteria gt fok file doc no_return Returns list of uuids of files that matches to criteria passed as FileCriteria Current implemen tation does not support specifying ctime and mtime at the same time other combinations of criterias are supported get_descriptor Fd fd gt fok fd_doc ferror any no_return Gets file descriptor from DB Argument should be uuid of file_descriptor record Non error return value is always ok veil_document record file_descriptor See dao save_record l and dao get record I for more details about veil_document wrapper Should not be used directly use dao handle 2 instead See dao handle 2 for more details get file File file gt fok file_doc ferror any no_return Gets file from DB Argument should be file see dao_types hrl for more details Should not be used directly use dao handle 2 instead See dao handle 2 for more details get file meta Fd fd gt fok fd_doc ferror any no_return Gets file meta from DB Argument should be uuid
43. that will end up handling REST requests Function Index route l 128 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Function Details route binary gt atomO binary noindex This function returns handler module and resource ID based on REST request path The argument is a list of binaries result of splitting request subpath on Subpath is all that occurs after rest in request path Should return a tuple the module that will be called to handle requested REST resource atom resource id or undefined if none was specified binary or atom undefined or undefined if no module was matched Submodules Handlers Documents file_transfer_handler Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This module processes file download and upload requests After vali dating them it conducts streaming to or from a socket or makes nitrogen display a proper error page Function Index e get_download_buffer_size 0 handle rest upload 3 handle upload request 2 e maybe handle request I Function Details multipart data Req headers cowboy http headers Req body binary Req I fend of part eof Req e Req req Asserts the validity of mutlipart POST request and proceeds with parsing and writing its data to a file at specified path Returns true for successful upload anf false otherwise handle upload requ
44. the MIT License Descritpion This file contains Nitrogen website code page rules viewer Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page shared files Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page system state Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page validate login Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code web 404 Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code 11 4 Veil modules 143 VeilFS Documentation Release 1 0 Submodules Elements Documents bootstrap_button Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains element definition for bootstrap compliant button Function Index e reflect 0 render_element l Function Details bootstrap checkbox Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains element definition for bootstrap compliant checkbox Function Index reflect 0 render_element l
45. turned access 4 checks if the calling process has specified by _mask premissions to file with given path Most storage helpers will always return 0 success therefore this method can be used only to check if calling process does NOT have permissions to the file chmod _sh_name string _sh_args string _path string mode inte ger ErrorCode integer error NIF not loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode re turn value shall be 0 if operation was succesfull otherwise negated POSIX error code will be returned chmod 4 changes file s mode chown sh name string sh args string _path string _uid integer gid integer ErrorCode integer error NIF not loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode re turn value shall be 0 if operation was succesfull otherwise negated POSIX error code will be returned chown 5 changes file s uid and gid chown_name _sh_name string _sh_args string path string _un ame string _gname string gt ErrorCode integer error NIF not loaded First 2 arguments of this method should come from storage_helper_info record Th
46. void addScheduler boost shared_ptr lt JobScheduler gt injected Sets JobScheduler object void setConfig boost shared ptr lt Config gt injected Sets Config object void setConnectionPool boost shared_ptr lt SimpleConnectionPool gt injected void staticDestroy Private Members std string m_root Filesystem root directory 2 12 veil client VeilFS 35 VeilFS Documentation Release 1 0 uid_t m_uid Filesystem owner s effective uid gid tm gid Filesystem owner s effective gid uid tm ruid Filesystem root real uid gid tm rgid Filesystem root real gid boost shared_ptr lt FslogicProxy gt m fslogic FslogicProxy instance boost shared_ptr lt StorageMapper gt m storageMapper StorageMapper instance boost shared_ptr lt MetaCache gt m_metaCache MetaCache instance boost shared_ptr lt helpers StorageHelperFactory gt m_shFactory Storage Helpers Factory instance std map lt std string std pair lt std string time_t gt gt m_linkCache Simple links cache ReadWriteLock m_linkCacheLock Private Static Attributes ReadWriteLock m_schedulerPoolLock std list lt boost shared_ptr lt JobScheduler gt gt m_jobSchedulers JobScheduler instances boost shared_ptr lt Config gt m_config Config instance boost shared_ptr lt SimpleConnectionPool gt m_connectionPool boost shared_ptr lt PushListener gt m_pushListener 36 Chapter 2 Classes CHAPTER 3
47. 107 11 1 veilcluster_node APP 4 4 ebb oe a A ee ee eee ke eae MG 107 112 Fen cluster node SU pl vade to Bech Gee de ht ay ea Be e Ses dan eds Nesa 107 11 3 Cluster Clements ir e eee een eee Sabha eh rap da eb doe 108 11 4 Veil modules IT MIT License VeilFS Documentation Release 1 0 Contents 1 VeilFS Documentation Release 1 0 2 Contents Part I Vell Client VeilFS Documentation Release 1 0 Documents VeilFS Documentation Release 1 0 CHAPTER 1 About VeilClient is a part of a meta file system called VeilFS which unifies access to different storage systems and provides a POSIX compatible interface 1 1 Goals The main goal of VeilClient is to provision a file system to different heterogeneous storage systems which will work in the user space e g Lustre GPFS DPM RODS VeilClient intends to reduce the complexity of accessing various storage systems by providing a standard POSIX compatible interface Furthermore storage systems connected to VeilClient can be geographically distributed and operated by different organizations The end user may operate on data from the storage systems as if they were stored at a local file system 1 2 Getting Started VeilClient is built with CMake More informations about compiling the project in Compilation section Sources are put in src The src includes subdirectories fuse and helpers T
48. 7 fslogicProxy h Author Beata Skiba Rafal Slota Copyright C 2013 ACK CYFRONET AGH This software is released under the MIT Licence Defines RENEW_LOCATION_MAPPING_TIME How many secs before expiration should focation mapping be renewed GET_FILE_ATTR GET_FILE_LOCATION GET_NEW_FILE_LOCATION 48 Chapter 3 Files VeilFS Documentation Release 1 0 RENEW_FILE_LOCATION GET_FILE_CHILDREN FILE_CHILDREN FILE_ATTR FILE_LOCATION CREATE_DIR DELETE_FILE FILE_NOT_USED RENAME_FILE CHANGE_FILE_PERMS FILE_LOCATION_VALIDITY CREATE_LINK GET_LINK LINK_INFO FUSE_MESSAGE ATOM FSLOGIC ACTION_NOT_ALLOWED 3 7 fslogicProxy h 49 VeilFS Documentation Release 1 0 ACTION_FAILED namespace veil namespace client class FslogicProxy The FslogicProxy class This class provides prox y methods that runs their correspondent cluster fslogic methods Each object of FslogicProxy has its own connection pool so in order to always use already opened connections there should be only one instance of this class unless you really need connection isolation between successive cluster requests Public Functions FslogicProxy FslogicProxy bool getFileAttr std string logicName protocol fuse_messages FileAttr amp attr Downloads file attributes from cluster bool getFileLocation std string logicName protocol fuse_messages FileLocation amp location Downloads file loc
49. Copyright This software is released under the MIT License Descritpion It is the behaviour of each application component All components de fined in subdirectories of veil_modules must implement it Function Index behaviour inforl Function Details behaviour info Arg Result Arg callbacks Other Fun def tuple e Other any e Result Fun def undefined Defines behaviour Submodules 11 4 7 Central logger Documents central_logger Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This module implements worker_plugin_behaviour to provide central logging functionalities Behaviours worker_plugin_behaviour Function Index e cleanup 0 e handle 2 init l 11 4 Veil modules 125 VeilFS Documentation Release 1 0 Function Details cleanup Result Error timeout term Result ok error Error worker plugin behaviour callback cleanup O Reconfigures lager back to standard handle ProtocolVersion term Request Result Error term Message string Metadata list Request ping get version subscribe Subscriber unsubscribe Subscriber dis patch log Message Timestamp Severity Metadata Response term Result ok ok Response I error Error pong Version Severity atom Subscriber pid Timestamp term Version term worker plugin behaviour callback handle 1 init Args
50. EST handler and what functions should be used to process the requests content types provided req state gt binary regO state 11 4 Veil modules 131 VeilFS Documentation Release 1 0 Cowboy callback function Returns content types that can be provided for the request Will call content types provided 112 from rest module behaviour delete resource req state term req0 state Cowboy callback function Handles DELETE requests Will call delete 2 from rest_module_behaviour get resource req state termO req0 state Cowboy callback function Handles GET requests Will call get 112 from rest_module_behaviour handle json data req state boolean req state Function handling application json requests handle multipart data req state booleanQ req state Function handling multipart form data requests handle_urlencoded_data req state boolean req state Function handling application x www form urlencoded requests init any any any upgrade protocol cowboy rest Cowboy callback function Imposes a cowboy upgrade protocol to cowboy_rest this module 1s now treated as REST module by cowboy resource_exists req state boolean req0 state Cowboy callback function Determines if resource identified by URL exists Will call exists 2 from rest module behaviour
51. Functions bool validateProxyConfig bool validateProxyCert std string getProxyCertPath std string getClusterHostname Variables bool debug 3 8 gsiHandler h 53 VeilFS Documentation Release 1 0 3 9 lock h Author Rafal Slota Copyright C 2013 ACK CYFRONET AGH This software is released under the MIT Licence Defines FAIRNESS_LEVEL Defines how many readers minimum should be processed after each writer which has priority MUTEX X Constructs name for mutex DECLARE_MUTEX_FOR X Declare mutex with given name INIT_MUTEX_FOR X Initialize mutex with given name as PTHREAD MUTEX RECURSIVE namespace veil namespace client Enums LockType enum See AutoLock AutoLock Values e READ LOCK e WRITE LOCK class ReadWriteLock The ReadWriteLock class Solves the library problem fast thread safe read access ReadWrite Lock should not be used directly Use AutoLock wrapper instead for automatic lock release using RAI Public Functions ReadWriteLock ReadWriteLock 54 Chapter 3 Files VeilFS Documentation Release 1 0 void readLock Locks for read operation void readUnlock Unlocks read lock void writeLock Locks for write operation void writeUnlock lt Unlocks write lock Private Functions DECLARE_MUTEX_FOR m_class DECLARE MUTEX FOR m res Private Members pthread cond tm classCond int m readers How many readers are i
52. ING FROM OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE 185
53. LE _vout FILE _verr char const _pass int passLen This function replaces standard main of grid proxy init tool First two arguments are standard main s arugments The others are used to capture standard output and standard error steams and pass cached certificate passphrase Variables int quiet int debug FILE vout stdout and stderr substitutions used in grid proxy utils FILE verr char const certPasswd int passwdLen char LONG_USAGE 3 18 gsi_utils h Author 70 Chapter 3 Files VeilFS Documentation Release 1 0 Rafal Slota Copyright C 2013 ACK CYFRONET AGH This software is released under the MIT Licence Functions int proxy_init int argc char argv FILE _vout FILE _verr char const _pass int passLen This function replaces standard main of grid proxy init tool First two arguments are standard main s arugments The others are used to capture standard output and standard error steams and pass cached certificate passphrase int proxy info int argc char argv FILE _vout FILE _verr This function replaces standard main of grid proxy info tool First two arguments are standard main s arugments The others are used to capture standard output and standard error steams Variables FILE vout stdout and stderr substitutions used in grid proxy utils FILE verr 3 18 gsi_utils h 71 VeilFS Documentation Release 1 0
54. See int fsyne const char path int datasync struct fuse file info fi fsync FUSE callback Not implemented yet See int setxattr const char path const char name const char value size_t size int flags setxattr FUSE callback Not implemented yet See int getxattr const char path const char name char value size_t size getxattr FUSE callback Not implemented yet See int listxattr const char path char list size_t size listxattr FUSE callback Not implemented yet See int removexattr const char path const char name removexattr FUSE callback Not implemented yet 62 Chapter 3 Files VeilFS Documentation Release 1 0 See int opendir const char path struct fuse_file_info filelnfo opendir FUSE callback Not implemented yet See int readdir const char path void buf fuse_fill_dir_t filler off_t offset struct fuse file info filelnfo readdir FUSE callback See int releasedir const char path struct fuse file info filelnfo releasedir FUSE callback Not implemented yet See int fsynedir const char path int datasync struct fuse_file_info filelnfo fsyncdir FUSE callback Not implemented yet See int init struct fuse_conn_info conn init FUSE callback See bool runTask TaskID taskId std string argO std string arg1 std string arg3 Task runner derived from Schedulable See ISchedulable runTas
55. Simple links cache ReadWriteLock m linkCacheLock Private Static Attributes ReadWriteLock m schedulerPoolLock std list lt boost shared_ptr lt JobScheduler gt gt m_jobSchedulers JobScheduler instances boost shared_ptr lt Config gt m_config Config instance boost shared_ptr lt SimpleConnectionPool gt m_connectionPool boost shared_ptr lt PushListener gt m pushListener Globus proxy utils 3 12 fmemopen c Typedefs typedef struct fmem fmem_t 3 12 fmemopen c 65 VeilFS Documentation Release 1 0 Functions int readfn void handler char buf int size int writefn void handler const char buf int size fpos_t seekfn void handler fpos_t offset int whence int closefn void handler FILE fmemopen void buf size_t size const char mode A BSD port of the fmemopen Linux method using funopen man docs for fmemopen man docs for funopen This method is ported from ingenuitas python tesseract project You must call fclose on the returned file pointer or memory will be leaked param buf The data that will be used to back the FILE methods Must be at least c size bytes param size The size of the c buf data param mode The permitted stream operation modes returns A pointer that can be used in the fread fwrite fseek fclose family of metho If a failure occurred NULL will be returned class fmem Public Members size t pos size_t size char buffer
56. VeilFS Documentation Release 1 0 VeilFS January 26 2014 Contents I VeilClient 3 1 About 7 L Gods ce seg Sek KR NN 7 12 Gemine Started 244 a caos AA F es be Eee ee OSS amp F es 7 13 Prerequisies ias a RE ed Sete SAG BA pe SAG Des 7 14 Compilation 2 2 ov ea ske bb bea ee bek A Gr eee d an 8 15 Usimg VellClient ces ss se se sek KS s Gp Se AS pre bar ik ig s 9 16 SUpport s ss se sd redd oe Ge Ge AE SERGE EEE G eik Gas Gere ae ae a Y 10 2 Classes 11 ZA el chent Autokodk eirag ae oR Bek ee A A LE A 11 22 velluclientiCOMIE os 4 kay see HA ee eG RG eke SEP RO SE A eS 12 23 velselhent FslosicProsy ma pa tse eS ed E Aha eed FG BAe ayes tek Gabe 16 24 veikichent ISchedulable csi a Sh a a gt a Bee el Bee 19 2 3 Vvell clhent JobScheduler sex keg ks God Gs aske sak wae A ee eat A 20 2 6 Vell chent MessageBuilder 2 ops sea KP A le EE amp Gee SO OS ela te 21 2 1 Vellschent MetaCache tirita ra A ba 8 AEG 22 2 8 gt Vell chent PushListeners vasse A RS ae RS Sake slake daa eS 24 29 velk chent ReadWiiteLock imss seg ba eee wae eee eee an S 25 2410 veilselient StoraseMapper Liers mi ai Kars dd best 27 2 11 veil cli nti VENBXceptuon as sag sa Sv Sark a GS Skee GS ed 29 212 vellscliente VSILES gt fede 09 F sd E BR P SE De FA SALE SNE 0 ena 30 3 Files 37 3 1 PgSIHandleree v s ag skades Geek a sed g te Ge ra see GE sed 37 3 2 jobSchedulefce o sa rau e a
57. ame string gt Result Attributes term ErrorDetail term ErrorGeneral atom e Result ok Attributes ErrorGeneral ErrorDetail Returns file attributes getfilelocation File term Result ErrorDetail term ErrorGeneral atom e Helper term Id term e Result ok Helper Id ErrorGeneral ErrorDetail Gets file location from fslogic or from cache File can be string path or uuid UUID ls DirName string ChildrenNum integer Offset integer Result 11 4 Veil modules 173 VeilFS Documentation Release 1 0 ErrorDetail term ErrorGeneral atom e FilesList list e Result ok FilesList ErrorGeneral ErrorDetail Lists directory uses data from db mkdir DirName string Result ErrorDetail term ErrorGeneral atom e Result ok ErrorGeneral ErrorDetail Creates directory in db mv From string To string Result ErrorDetail term ErrorGeneral atom e Result ok ErrorGeneral ErrorDetail Moves directory in db read File term Offset integer Size integer gt Result e Bytes binary ErrorDetail term ErrorGeneral atom e Result ok Bytes ErrorGeneral ErrorDetail Reads file uses logical name of file First 1t gets information about storage helper and file id at helper Next it uses storage helper to read data from file File can be
58. amefhost main ccm main ccm nodefhost opt_ccms opt_ccm_nodel host opt_ccm_node2 hc The expression after name specifies the node for which the release will be generated It can be one of CCMs listed later on The expression after main ccm specifies the node name of the main CCM The expression after opt_ccms specifies the list of optional CCMs These arguments are not manda tory The expression after db_nodes specifies the list of all DBMS nodes 88 Chapter 7 Development VeilFS Documentation Release 1 0 The above argument string can be either placed in file gen_dev args located in the root directory or passed to Makefile Optionally one of following can be added to arguments no compile gt skips code compilation straight to release generation no generat gt skips code compilation and release generation creates a copy of existing rel 7 5 2 Generating a release make gen test node args name node namefhost main ccm main ccm nodefhost opt_ccms opt ccn make gen test node from file After either of these operations the release will be placed in releases test cluster node name with all the environment variables set up 7 5 3 Starting and managing generated packages make start node node node name gt starts the node called node name as a da make attach to node node node name gt attaches to the running node called node_ make start node console
59. anymore std string changeFilePerms std string path mode_t mode Change file permissions std string createLink std string from std string to Creates symbolic link from to file to std pair lt std string std string gt getLink std string path Gets path pointed by link 2 3 veil client FslogicProxy 17 VeilFS Documentation Release 1 0 std string updateTimes std string path time_t atime 0 time_t mtime 0 time_t ctime 0 Updates time meta attributes for specific file std string changeFileOwner std string path uid_t uid std string uname fer Updates file s owner std string changeFileGroup std string path gid t gid std string gname 2 Updates file s group owner void pingCluster std string nth bool runTask TaskID taskId std string arg0 std string arg1 std string arg3 Task runner derived from Schedulable See ISchedulable runTask Protected Functions bool sendFuseReceiveAnswer const google protobuf Message amp fMsg google protobuf Message amp response Sends and receives given protobuf message High level method used to send serialized protobuf message to cluster and return its response as given by reference object Both message and response types have to be subtype of FuseMessage Return true only if received response message is initialized see google protobuff Message IsInitialized std string sendFuseRece
60. apping See StorageMapper m_fileMapping boost shared_ptr lt FslogicProxy gt m fslogic Reference to FslogicProxy instance 28 Chapter 2 Classes VeilFS Documentation Release 1 0 See VeilFS m_fslogic 2 11 veil client VeilException class veil client VeilException Base class of all VeilClient exceptions Public Functions VeilException Default empty constuctor VeilException std string veilError std string logMsg VeilException main constructor See VeilFS translateError Parameters veilError POSIX error name Should be compatibile with VeilFS translateError e logMsg Human readable excpetion reason message VeilException const char what Returns VeilException m_logMessage std string veilError Returns VeilException m_veilError Private Members std string m_logMessage Human readable excpetion reason message std string m_veilError POSIX error name This error string should be compatibile with VeilFS translateError 2 11 veil client VeilException 29 VeilFS Documentation Release 1 0 See VeilFS translateError 2 12 veil client VeilFS class veil client VeilFS The VeilF S main class This class contains FUSE all callbacks so it basically is an heart of the filesystem Technically VeilFS is an singleton created on programm start and registred in FUSE daemon Public Functions VeilFS std string path boost shared_ptr
61. application json Location redirect link to shared file Status Codes 303 See Other 422 Unprocessable Entity 500 Internal Server Error An example curl http curl haxx se request to share file dir file txt would be curl i k cert proxy cert H content type application json X POST https example Example request POST rest latest files shares HTTP 1 1 Host example com Header content type application json Data dir file txt Example responses HTTP 1 1 303 See Other connection close server Cowboy date Sun 05 Jan 2014 18 38 17 GMT content length 0 Access Control Allow Origin content type application json location https example com share 04ef3d726a2554f240bbl5bf4cfa2al3 9 4 6 rest latest shares guid Methods GET rest latest shares guid Retrieve shared file details as a record of structure property value Fields of returned record eshared file path eshared file download url 9 4 Resources 101 VeilFS Documentation Release 1 0 Parameters guid string shared files globally unique identifier Response Headers Content Type application json Status Codes 200 OK 404 Not Found 500 Internal Server Error Example request GET rest latest files shares 04ef3c62ea0cdba9cd2ac1a860835efe HTTP 1 1 Header content type application json Host example com Example responses HTTP 1 1 200 OK connection close serv
62. are validated using Grid Security Infras tructure If sent certificate fails verification or is not not present in database connection is closed If sent certificate passes verification its distinguished name is associated with single entry in users database and since than client can access cluster resources How to create proxy certificate To create proxy certificate run grid proxy init script with argument out path_to_proxy_cert This script can be found at root directory on cluster and requires presence of main PLGrid certificate in root globus directory Example sh grid proxy init out proxy_cert How to use proxy certificate Proxy certificate can be used with any computer software that provides data trans fer using HTTPS protocol and supports HTTPS certificates In following exam ple content of dir directory will be retrieved using HTTP GET request and curl http curl haxx se tool Example curl i k cert proxy_cert X GET https example com rest latest files dir 95 VeilFS Documentation Release 1 0 Warning VeilFS REST API doesn t work with every version of curl tool It has to be linked to OpenSSL library To check it use curl version command 9 2 Errors Validation If a request fails due to client error the resource will return an HTTP response code in the 40x range These can be broadly categorised into HTTP Code Description 400 Bad Request 401 Unauthorized 403 F
63. ariable Protected Functions template lt typename T gt T get std string opt Internal implementation of Config getValue 14 Chapter 2 Classes VeilFS Documentation Release 1 0 See Config getValue template lt typename T gt T getValue std string opt Returns type specialized value of given config option bool runTask TaskID taskId std string argO std string arg1 std string arg3 Task runner derived from Schedulable See ISchedulable runTask void setupDefaults Protected Attributes bool defaultsLoaded ReadWriteLock m_access std string m_globalConfigPath Path to global config file See Config setGlobalConfigFile std string m_userConfigPath Path to user config file See Config setUserConfigFile YAML Node m_globalNode Global config object YAML Node m_userNode User config object YAML Node m_envNode Temp config object used to manipulate env settings YAML Node m_defaultsNode 2 2 veil client Config 15 VeilFS Documentation Release 1 0 Default configs Protected Static Attributes std string m_requiredOpts Array containing required options names std string m_envCWD Saved CWD env variable std string m_envHOME Saved HOME env variable std map lt std string std string gt m_envAll All saved env variables 2 3 veil client FslogicProxy class veil client FslogicProxy The FslogicProxy class This class provides proxy methods that ru
64. ass veil client MessageBuilder The MessageBuilder class This class can be used to build protobuf messages used to communicate with cluster Theres encode and decode method for each base message type used by VeilClient Arguments matches proto specification of their messages Public Functions MessageBuilder MessageBuilder 2 6 veil client MessageBuilder 21 VeilFS Documentation Release 1 0 protocol fuse_messages FuseMessage createFuseMessage std string id std string messageType std string messageInput protocol communication_protocol ClusterMsg createClusterMessage std string moduleName std string messageType std string answerType std string answerDecoderName bool synch std string input protocol communication_protocol ClusterMsg createClusterMessage std string moduleName std string messageType std string answerType std string answerDecoderName bool synch protocol communication_protocol ClusterMsg packFuseMessage std string messageType std string answerType std string answerDecoderName std string messagelnput protocol fuse_messages FuseMessage decodeFuseAnswer protocol communication_protocol Answer amp answer std string decodeAtomAnswer protocol communication_protocol Answer amp answer 2 7 veil client MetaCache class veil client MetaCache Class responsible for caching file attributes 22 Chapter 2 Classes VeilFS Documentation Releas
65. ate Timeout reply Reply NewState hibernate noreply NewState noreply NewState Timeout noreply NewState hibernate stop Reason Reply NewState stop Reason NewState Timeout non_neg_integer I infinity gen server handle call 3 handle cast Request term State term gt Result NewState term Result noreply NewState noreply NewState Timeout noreply NewState hiber nate stop Reason term NewState Timeout non_neg_integer infinity gen_server handle_cast 2 handle_info Info timeout term State term Result NewState term Result noreply NewState noreply NewState Timeout noreply NewState hiber nate stop Reason term NewState 120 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Timeout non_neg_integer infinity gen_server handle_info 2 init Args term Result e Result ok State fok State Timeout ok State hibernate stop Reason term ignore State term Timeout non_neg_integer infinity gen_server init l send_to_fuse Fuseld string Message term MessageDecoder string gt Result e Result callback_node_not_found node_manager_error dispatcher_error ok term Sends message to fuse start link Result e Error already_started Pid term O e Pid pid e Result ok Pid ignore error Error Sta
66. ation info bool getNewFileLocation std string logicName mode_t mode protocol fuse_messages FileLocation amp location Query cluser to create new file in DB and get its real location int renewFileLocation std string logicName Try to renew location validity for given file 50 Chapier 3 Files VeilFS Documentation Release 1 0 bool getFileChildren std string dirLogicName uint32_t children_num uint32_t offset std vector lt std string gt amp childrenNames List files in given folder std string renameFile std string fromLogicName std string toLogicName Rename move file to new location std string createDir std string logicName mode_t mode Create directory std string deleteFile std string logicName Delete given file bool sendFileNotUsed std string logicName Inform cluster that file isnt used anymore std string changeFilePerms std string path mode_t mode Change file permissions std string createLink std string from std string to Creates symbolic link from to file to std pair lt std string std string gt getLink std string path Gets path pointed by link std string update Times std string path time_t atime 0 time_t mtime 0 time_t ctime 0 Updates time meta attributes for specific file 3 7 fslogicProxy h 51 VeilFS Documentation Release 1 0 std string changeFileOwner std string path uid_t uid std
67. atom e Result boolean ErrorGeneral ErrorDetail Checks if file exists get_ets_name Result e Result atom Generates name of ets table for proc get file by uuid UUID string gt Result ErrorDetail term ErrorGeneral atom File term O e Result ok File ErrorGeneral ErrorDetail Gets file record on the basis of uuid get file full name by uuid UUID string Result ErrorDetail term 172 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 ErrorGeneral atom e FullPath string O e Result ok FullPath ErrorGeneral ErrorDetail Gets file full name with root of the user s system on the basis of uuid get file name by uuid UUID string Result ErrorDetail term ErrorGeneral atom e Name term e Result ok Name ErrorGeneral ErrorDetail Gets file name on the basis of uuid get file user dependent name by uuid UUID string gt Result ErrorDetail term ErrorGeneral atom e FullPath string e Result ok FullPath ErrorGeneral ErrorDetail Gets file full name relative to user s dir on the basis of uuid get share Key file File uuid fuser User uuid fuuid UUID uuid Result ErrorDetail term ErrorGeneral atom e Result ok Share doc ErrorGeneral ErrorDetail Share doc term Gets info about share from db getfileattr FileN
68. b_proc Name atom MaxDepth integer MaxWidth integer ProcFun term MapFun term Result e Result pidO Starts sub proc stop Plugin ok e Plugin atom 11 3 Cluster elements 115 VeilFS Documentation Release 1 0 Stops the server terminate Reason State term gt Any term e Reason normal shutdown shutdown term term gen_server terminate 2 Submodules 11 3 4 Request dispatcher Documents gsi_ handler Authors Rafal Slota Copyright This software is released under the MIT License Descritpion This module manages GSI validation Function Index call 3 find_eec_cert 3 init O is_proxy_certificate 1 load_certs l proxy_subject l update_crls l verify_callback 3 Function Details call Module atom Method atom Args term gt ok no_return Calls apply Module Method Args on one of started slave nodes If slave node is down initializes restart procedure and tries to use another node However is all nodes are down error is returned and GSI action is interrupted e g peer verification fails find eec cert CurrentOtp OTPCertificate Chain OTPCertificate IsProxy boolean gt ok OTPCertificate no_return For given proxy certificate returns its EEC init gt ok Initializes GSI Handler This method should be called once before using any other method from this module is_
69. box 2 9 veil client ReadWriteLock class veil client ReadWriteLock 2 9 veil client ReadWriteLock 25 VeilFS Documentation Release 1 0 The ReadWriteLock class Solves the library problem fast thread safe read access ReadWriteLock should not be used directly Use AutoLock wrapper instead for automatic lock release using RAI Public Functions ReadWriteLock ReadWriteLock void readLock Locks for read operation void readUnlock Unlocks read lock void writeLock Locks for write operation void writeUnlock lt Unlocks write lock Private Functions DECLARE MUTEX FOR m class DECLARE MUTEX FOR m res Private Members pthread cond tm classCond int m readers How many readers are in critical section right now 26 Chapter 2 Classes VeilFS Documentation Release 1 0 int m writers How many writers are waitning in queue to access critical section int m fairness How many readers was in critical section since last writer left 2 10 veil client StorageMapper class veil client StorageMapper Public Functions StorageMapper boost shared_ptr lt FslogicProxy gt fslogicProxy StorageMapper std pair lt locationInfo storagelnfo gt getLocationInfo std string logical_name bool useCluster false Gets file location information along with storage info for storage heleper s calls Return std pair of locationinfo and storagelnfo struc
70. ditions 140 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 get requested hostname gt string Returns the hostname requested by the client get user dn gt string Returns user s DN retrieved from his session state logotype footer MarginTop integer gt list Convienience function to render logotype footer coming after page content storage defined gt boolean Checks if any storage is defined in the database top menu ActiveTabID any gt list Convienience function to render top menu in GUI pages Item with ActiveTabID will be high lighted as active top menu ActiveTabID any SubMenuBody any gt list Convienience function to render top menu in GUI pages Item with ActiveTabID will be high lighted as active Submenu body list of nitrogen elements will be concatenated below the main menu user logged in gt boolean Checks if the client has a valid login session page about Authors Krzysztof Trzepla Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page contact support Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code page error Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains Nitrogen website code 11 4 Veil modules
71. dule atom Method atom Args Arg term term Error e Error error rpc_retry_limit_exceeded ferror term Same as call 2 but with custom Module delete Host atom ok error timeout Deletes db host name from store host pool insert Host atom ok error timeout Inserts db host name into store host pool reactivate Host atom ok error no host error timeout Reactivate banned db host name If given Host wasn t banned nothing happens and will return ok If given Host wasn t inserted returns error no host store exec sequential Msg term ok I error Error term Executes Msg Caller must ensure that this method won t be used concurrently Currently this method is used as part of internal module implementation although it has to be exported because it s called by gen server which ensures it s sequential call dao json Authors Rafal Slota Copyright This software is released under the MIT License Descritpion BigCouch document management module very simple BigCouch specific JSON creator 11 4 Veil modules 153 VeilFS Documentation Release 1 0 Function Index get_field 2 get_fields l mk_bin l mk_doc l mk_field 3 mk_fields 3 mk obj 0 mk_str l reverse_fields l rm_field 2 rm_fields 2 Function Details get_field DocOrObj Name string anyO I error not found I error in valid object
72. e Descritpion This module coordinates central cluster Behaviours gen_server http www erlang org doc man gen_server html RestartStrategy one_for_all one_for_one rest_for_one simple one for one ChildSpec Id termQ StartFunc RestartPolicy Type worker supervisor Modules SupervisionPolicy RestartStrategy MaxR non neg integer MaxT 108 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Function Index code_change 3 handle_call 3 handle_cast 2 handle_info 2 init 1 monitoring loop l monitoring loop 2 start link 0 start link l start monitoring loop 2 stop 0 terminate 2 Function Details code change OldVsn State term Extra term Result e OldVsn Vsn down Vsn e Result ok NewState term I ferror Reason termQ Vsn term gen_server code_change 3 handle call Request term From pid Tag term State term gt Result NewState term Reason term Reply term Result reply Reply NewState reply Reply NewState Timeout reply Reply NewState hibernate noreply NewState noreply NewState Timeout noreply NewState hibernate I stop Reason Reply NewState stop Reason NewState Timeout non_neg_integer infinity gen_server handle_call 3 handle_cast Request term State term Result NewState term e Result noreply NewState noreply N
73. e 1 0 See VeilFS getattr Public Functions MetaCache MetaCache void addAttr std string path struct stat amp attr Cache given attributes Expiration time can be set using configuration file void clearAttrs Clear whole cache bool getAttr std string path struct stat attr Gets file attributes from cache Return Bool saying if operation succeed and stat struct was filled with data Parameters e stat Pointer to stat structure that should be filled with data from cache void clearAttr std string path Remove cache for given file bool updateTimes std string path time_t atime 0 time_t mtime 0 time_t ctime 0 Update time meta attributes for specific file in cache Returns true if cache was updated or false if given file wasn t found in cache bool updateSize std string path size_t size 2 7 veil client MetaCache 23 VeilFS Documentation Release 1 0 Update size meta attribute for specific file in cache Returns true if cache was updated or false if given file wasn t found in cache bool runTask TaskID taskId std string argO std string arg1 std string arg3 Task runner derived from Schedulable See ISchedulable runTask Protected Attributes std map lt std string std pair lt time_t struct stat gt gt m_statMap This is the cache map Value of this std map is std pair containing expiration time of attributes and stat struct itself ReadWriteL
74. e following 1 ResponseBody of the same type as content_types_provided 2 returned for this request 2 Cowboy type stream function serving content of the same type as content_types_provided 2 returned for this request 3 halt atom if method is not supported handle_multipart_data req binary binary term gt booleanQ req Optional callback to handle multipart requests Data should be streamed in handling module with use of cowboy_multipart module Method can be lt gt or lt gt Should handle the request and return true false indicating the result Should always return false if the method is not supported post req binary binary term boolean true binary regO 11 4 Veil modules 135 VeilFS Documentation Release 1 0 Will be called for POST request after the request has been validated Should handle the request and return true false indicating the result Should always return false if the method is not supported Returning true URL will cause the reply to contain 201 redirect to given URL put reg binary binary term boolean reqQ Will be called for PUT request on given ID after the request has been validated Should handle the request and return true false indicating the result Should always return false if the method 1s not supported validate req binary binary term boolean reqO Should return true false dependin
75. e returned as a proplist and later used to authenticate a user in the system Must be called from within nitrogen page context to work precisely from openid redirection page rest_utils Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This module provides convinience functions designed for REST han dling modules Function Index e decode_from_json 1 e encode_to_json l e map 2 e unmap 3 Function Details decode from json term gt binary Convinience function that convert JSON binary to an erlang term encode to json term gt binary Convinience function that convert an erlang term to JSON producing binary result Possible terms can be nested struct Props Props is a structure as a proplist e g id 13 mes sage mess Props alias for above array Array Array is a list e g 13 mess map record atom gt tuple0 Converts a record to JSON conversion ready tuple list For this input RecordToMap some_record id 123 message mess Fields id message The function will produce id 123 message mess NOTE rest utils hrl contains convienience macro that will in clude this function in code with Fields listed for given record name unmap tuple record atom 1 gt tupleQ Converts a tuple list resulting from JSON to erlang translation into an erlang record Reverse process to map 2 For this inpu
76. ect condition Current implementeation supports fallowing selects by valid to Time select all records whose valid to field is gt Time Should not be used directly use dao handle 2 instead remove connection info SessID uuid gt ok no return Removes connection info record with given SessID form DB Should not be used directly use dao handle 2 instead remove fuse session Fuseld uuid gt ok no return Removes fuse session record with given FuseID form DB and cache Should not be used directly use dao handle 2 instead save connection info connection_info veil_document gt ok Id string no return Saves connection info record to DB Should not be used directly use dao handle 2 instead save fuse session fuse_session veil_document gt fok Id string no return Saves fuse session record to DB If ftfuse session valid to field is not valid i e its value is less then current timestamp it will be set to default value specified in application config Should not be used directly use dao handle 2 instead save state Rec tuple gt fok Id string no return Saves cluster state Rec to DB with ID cluster_state Should not be used directly use dao handle 2 instead save state Id atom Rec tuple gt fok Id string no return Saves cluster state Rec to DB with ID Id Should not be used directly use dao handle 2 in
77. ed pid_not_found pid deleted Deletes callback get_callback State term Fuseld string Result e Result non pid Gets callback to fuse if there are more than one callback it chooses one handle call Request term From pid Tag term State term gt Result NewState term Reason term Reply term Result reply Reply NewState reply Reply NewState Timeout reply Reply NewState hibernate noreply NewState noreply NewState Timeout noreply NewState hibernate stop Reason Reply NewState stop Reason NewState Timeout non_neg_integer infinity gen_server handle_call 3 handle_cast Request term State term Result e NewState term e Result noreply NewState noreply NewState Timeout noreply NewState hiber nate stop Reason term NewState Timeout non_neg_integer infinity gen_server handle_cast 2 handle_info Info timeout term State term Result e NewState term e Result noreply NewState noreply NewState Timeout noreply NewState hiber nate stop Reason term NewState Timeout non_neg_integer infinity gen_server handle_info 2 init Args term Result Result ok State fok State Timeout ok State hibernate stop Reason term ignore State term Timeout non_neg_integer infinity gen_server init l
78. element tuple containing type of re source used to filter descriptors and resource itself Currently only by file File file is supported Second argument limits number of rows returned 3rd argument sets offset of query skips first Offset rows Non error return value is always ok veil_document record file_descriptor See dao save_record I and dao get_record I for more details about veil_document wrapper Should not be used directly use dao handle 2 instead See dao handle 2 for more details list_dir Dir file N pos integer Offset non_neg_integer ok file_doc Lists N files from specified directory starting from Offset Non error return value is always list of veil_document record file records Should not be used directly use dao handle 2 instead See dao handle 2 for more details list_storage gt ok storage_doc no_return Lists all storage docs Non error return value is always list of veil_document record stor age_info records Should not be used directly use dao handle 2 instead See dao handle 2 for more details lock file UserID string FileID string Mode write read gt not_yet_implemented Puts a read write lock on specified file owned by specified user Should not be used directly use dao handle 2 instead See dao handle 2 for more details Not yet implemented This is placeholder template method only remove_descriptor Fd fd
79. ent console loglevel get current loglevel gt integer Returns current loglevel as set in application s env get default loglevel gt integer Returns default loglevel as set in application s env get include stacktrace gt boolean Returns get include stacktrace env value loglevel atom to int LoglevelAsAtom atom gt integer Returns loglevel number associated with loglevel name loglevel int to atom LoglevelAsInt integer gt atom Returns loglevel name associated with loglevel number parse process info ProcessInfo tuple tupleQ Changes standard process info tuple into metadata proplist set console loglevel Loglevel integer atom ok I error badarg 118 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Changes current console loglevel to desired Argument can be loglevel as int or atom default atom can be used to set it back to default default is what is defined in sys config set include stacktrace boolean ok error badarg Changes include stacktrace env to true or false set loglevel Loglevel integer atom ok I error badarg Changes current global loglevel to desired Argument can be loglevel as int or atom default atom can be used to set it back to default should log LoglevelAsInt integer gt boolean Determines if logs with provided loglevel should be logged or discarded records translator
80. ents worker plugin behaviour to provide func tionality of remote files manager Behaviours worker plugin behaviour Function Index e cleanup 0 e handle 2 init l 11 4 Veil modules 123 VeilFS Documentation Release 1 0 Function Details cleanup ok worker plugin behaviour callback cleanup 0 handle ProtocolVersion term Request term Result e Result term worker plugin behaviour callback handle 1 Processes standard worker requests e g ping and requests from FUSE init Args term gt list worker plugin behaviour callback init 1 11 4 4 rtransfer Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module implements worker plugin behaviour to provide remote transfer manager functionality gateways management Behaviours worker_plugin_behaviour Function Index e cleanup 0 e handle 2 init l Function Details 11 4 5 rule manager Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module implements worker plugin behaviour to provide func tionality of rule engines manager update of rules in all types of rule engines Behaviours worker plugin behaviour Function Index e cleanup 0 e handle 2 e init l 124 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Function Details 11 4 6 worker_plugin_behaviour Authors Michal Wrzeszcz
81. equest 3 halt atom if method is not supported get req binary binary gt termO stream integer function halt req req Will be called for GET request with specified ID e g rest resource some_id Should return one of the following 1 ResponseBody of the same type as content_types_provided 2 11 4 Veil modules 137 VeilFS Documentation Release 1 0 returned for this request 2 Cowboy type stream function serving content of the same type as content_types_provided 2 returned for this request 3 halt atom if method is not supported post req binary binary term boolean I true binary regO Will be called for POST request after the request has been validated Should handle the request and return true false indicating the result Should always return false if the method is not supported Returning true URL will cause the reply to contain 201 redirect to given URL put reg binary binary term boolean reqQ Will be called for PUT request on given ID after the request has been validated Should handle the request and return true false indicating the result Should always return false if the method 1s not supported validate req binary binary term boolean reqQ Should return true false depending on whether the request is valid in terms of the handling module Will be called before POST or PUT should discard un
82. er Cowboy date Sun 05 Jan 2014 17 52 16 GMT content length 108 Access Control Allow Origin content type application json file path dir file txt download path https example com share 04ef3c62ea0cdba9 c DELETE rest latest shares guid Stop sharing existing file This request removes specified file from a list of shared files Parameters guid string shared files globally unique identifier Response Headers Content Type application json Status Codes 204 No Content 405 Method Not Allowed 500 Internal Server Error Example request DELETE rest latest files shares 04ef3c62ea0cdba9cd2acla860835efe HTTP 1 1 Header content type application json Host example com Example responses HTTP 1 1 204 No Content connection close server Cowboy date Sun 05 Jan 2014 17 58 05 GMT content length 0 Access Control Allow Origin content type application json 102 Chapter 9 REST API VeilFS Documentation Release 1 0 9 5 Structure of the REST URIs VeilFS s REST APIs provide access to resources data entities via URI paths To use a REST API your application will make an HTTP request and parse the response The VeilFS REST API uses JSON as its communication format and the standard HTTP methods like GET PUT POST and DELETE URIs for VeilFS s REST API resource have the following structure http host port rest api version path to resource Where api version can be
83. es file that was previously opened with open 4 rename _sh_name string _sh_args string _from string _to string ErrorCode integer error NIF not loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode re turn value shall be 0 if operation was succesfull otherwise negated POSIX error code will be returned rename 4 shall be used to rename move file from _from path to _to path rmdir _sh_name string _sh_args string _path string gt ErrorCode integer error NIF_not_loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode re turn value shall be 0 if operation was succesfull otherwise negated POSIX error code will be returned rmdir removes directory with given _path start Prefix string ok ferror Reason termQ This method loads NIF library into erlang VM This should be used once before using any other method in this module statfs sh name string _sh_args string _path string gt ErrorCode integer st_statvfs I error NIF_not_loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Hel
84. est Reg req record ok Params tuple Files fuploaded_file error incorrect_session Asserts the validity of mutlipart POST request and proceeds with parsing or returns an error Returns list of parsed filed values and file body maybe handle request Reg req gt boolean NewReq req Intercepts the request if its a user content or shared file download request or doeas nothinf otherwise Checks its validity and serves the file or redirects to error page Should be called on a request coming directly to nitrogen_handler 11 4 Veil modules 129 VeilFS Documentation Release 1 0 nagios_handler Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This module handles Nagios monitoring requests Function Index handle 2 init 3 terminate 3 Function Details handle term term ok termQ term Handles a request producing an XML response init any term any ok termo Cowboy handler callback no state is required terminate term term term ok Cowboy handler callback no cleanup needed nitrogen_handler Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This is the callback module for cowboy to handle requests by passing them to the nitrogen engine Function Index handle 2 init 3 terminate 3 Function Details handle Request Options
85. ethod for each base message type used by VeilClient Arguments matches proto specification of their messages Public Functions MessageBuilder MessageBuilder protocol fuse_messages FuseMessage createFuseMessage std string id std string messageType std string messageInput protocol communication_protocol ClusterMsg createClusterMessage std string moduleName std string messageType std string answerType std string answerDecoderName bool synch std string input 3 10 messageBuilder h 57 VeilFS Documentation Release 1 0 protocol communication_protocol ClusterMsg createClusterMessage std string moduleName std string messageType std string answerType std string answerDecoderName bool synch protocol communication_protocol ClusterMsg packFuseMessage std string messageType std string answerType std string answerDecoderName std string messagelnput protocol fuse messages FuseMessage decodeFuseAnswer protocol communication_protocol Answer amp answer std string decodeAtomAn swer protocol communication_protocol Answer amp answer 3 11 veilfs h Author Rafal Slota Copyright C 2013 ACK CYFRONET AGH This software is released under the MIT Licence Defines GLOBAL_CONFIG_FILE The name of default global config file CONFIG_ARGV_OPT_NAME The command line pattern used to find user config path ARGV DIR_BATCH_SIZE How many dirent should be fetch from cluste
86. ewState Timeout noreply NewState hiber nate stop Reason term NewState Timeout non_neg_integer infinity gen_server handle_cast 2 handle info Info timeout term State term Result 11 3 Cluster elements 109 VeilFS Documentation Release 1 0 e NewState term e Result noreply NewState I noreply NewState Timeout noreply NewState hiber nate stop Reason term NewState Timeout non_neg_integer infinity gen_server handle_info 2 init Args term Result e Result ok State fok State Timeout ok State hibernate stop Reason term ignore State term Timeout non_neg_integer infinity gen_server init l monitoring loop Flag ok e Flag on off Loop that monitors if nodes are alive monitoring loop Flag Nodes ok e Flag on off e Nodes list Beginning of loop that monitors if nodes are alive start link Result e Error already started Pid term O e Pid pidO e Result ok Pid ignore I error Error Starts cluster manager start link Mode Result e Error already started Pid term O e Mode test normal e Pid pidO e Result ok Pid ignore I error Error Starts cluster manager start monitoring loop Flag Nodes ok e Flag on off e Nodes list Starts loop that monitors if nodes are alive stop ok Stops the server terminate Reason State term
87. files_number user group UUID uuid string ProtocolVersion inte ger gt Result e Result ok Sum I error anyQ e Sum integer Returns number of user s or group s files get full file name FileName string Result e ErrorDesc atom e FullFileName string e Result ok FullFileName error ErrorDesc 11 4 Veil modules 167 VeilFS Documentation Release 1 0 Gets file s full name user s root is added to name but only when asking about non group dir get user id Result e ErrorDesc atom e Result ok UserID error ErrorDesc e UserID term Gets user s id get_user_root UserDoc term Result e ErrorDesc atom e Result ok RootDir error ErrorDesc RootDir string Gets user s root directory handle ProtocolVersion term Request term gt Result e Result term worker plugin behaviour callback handle l Processes standard worker requests e g ping and requests from FUSE init Args term gt list worker plugin behaviour callback init 1 fslogic storage Authors Rafal Slota Copyright This software is released under the MIT License Descritpion This module exports storage management tools for fslogic Function Index get sh for fuse 2 insert storage 2 insert storage 3 select storage 2 Function Details get sh for fuse FuselD string Storage storage_info stor age helper info Returns
88. fset struct fuse file info filelnfo int wrap write const char path const char buf size tsize off t offset struct fuse file info filelnfo int wrap_statfs const char path struct statvfs statInfo int wrap flush const char path struct fuse file info filelnfo int wrap release const char path struct fuse file info fileInfo 3 5 veilFuse cc 41 VeilFS Documentation Release 1 0 int wrap fsync const char path int datasync struct fuse_file_info fi int wrap readdir const char path void buf fuse_fill_dir_t filler off_t offset struct fuse_file_info filelnfo int wrap opendir const char path struct fuse_file_info filelnfo int wrap releasedir const char path struct fuse file info filelnfo int wrap_fsyncdir const char path int datasync struct fuse file info fileInfo int vfs opt proc void data const char arg int key struct fuse args outargs void oper init void fuse init int main int argc char argv char envp Variables boost shared ptr lt VeilFS gt VeilAppObject Main application object filesystem state 42 Chapter 3 Files VeilFS Documentation Release 1 0 Headers struct fuse_operations vfs oper 3 6 config h Defines CLUSTER_HOSTNAME_OPT Config option names CLUSTER_PORT_OPT LOG_DIR_OPT PEER_CERTIFICATE_FILE_OPT ENABLE_ATTR_CACHE_OPT ATTR_CACHE_EXPIRATION_TIME_OPT ENABLE_LOCATION_CACHE_OPT
89. g on whether the request is valid in terms of the handling module Will be called before POST or PUT should discard unprocessable requests rest module behaviour Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This behaviour should be implemented by modules handling REST re quests It ensures the presence of required callbacks Function Index behaviour inforl Function Details behaviour info Arg Result Arg callbacks Other e Fun def tuple e Other any e Result Fun def undefined Defines the behaviour lists the callbacks and their arity rest_shares Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This module implements rest_module_behaviour and handles all REST requests directed at rest shares path creating and deleting shares listing shares and retrieving share details Behaviours rest module behaviour 136 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Function Index allowed_methods 3 content_types_provided 2 content_types_provided 3 delete 3 exists 3 get 2 get 3 post 4 put 4 validate 4 Function Details allowed methods req binary binary gt binaryQ regO Should return list of methods that are allowed and directed at specific Id e g 1f Id undefined gt lt gt lt gt if Id undefined gt lt gt lt gt lt
90. gs workers workerl host main ccm main ccm nodefhost opt_ccms opt_c make start test env from file It is important that the same args are passed to Makefile or remain in the args file This is the way for the script to know which release packages need to be started 7 5 7 Generating and immediately starting the whole cluster make gen start test env args workers workerl host main ccm main ccm nodefhost opt_ccms c make gen start test env from file Every node can be started independently with use of start node attach to node and start node console make targets 7 5 8 Compilation and Releases from Sources To build a working release of VeilCluster from scratch type the following commands make test make generate After this step we should have a veil_cluster_node folder in the releases folder 7 5 9 Execution Now we are ready to start CCM and worker processes S cp R releases veil_cluster_node releases veil_cluster_node_worker releases veil cluster node bin veil cluster name ccm 127 0 0 1 main ccm ccm 127 0 0 1 c releases veil cluster node worker bin veil cluster name worker 127 0 0 1 main ccm ccm 12 curl https 127 0 0 1 8000 If everything went correct two processes should be started and a default web site of the VeilCluster should be available Note To have a fully working VeilFS installation we should also start a BigCouch instance
91. gt content types provided req binary gt binary0 regO Should return list of provided content types without specified ID e g rest resource Should take into account different types of methods PUT GET etc if needed Should return empty list if method is not supported content types provided req binary binary binary0 reqO Should return list of provided content types with specified ID e g rest resource some_id Should take into account different types of methods PUT GET etc if needed Should return empty list if method is not supported delete req binary binary gt boolean reqQ Will be called for DELETE request on given ID Should try to remove specified resource and return true false indicating the result Should always return false if the method is not supported exists req binary binary booleanQ reqQ Should return whether resource specified by given ID exists Will be called for GET PUT and DELETE when ID is contained in the URL get req binary gt term stream integer function halt regO req Will be called for GET request without specified ID e g rest resource Should return one of the following 1 ResponseBody of the same type as content_types_provided 1 re turned for this request 2 Cowboy type stream function serving content of the same type as content_types_provided l returned for this r
92. gt Any term e Reason normal shutdown shutdown term term 110 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 gen_server terminate 2 11 3 2 node_manager Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module is a gen_server that coordinates the life cycle of node It starts stops appropriate services according to node type and communicates with ccm if node works as worker Node can be ccm or worker However worker_hosts can be also started at ccm nodes Behaviours gen_server http www erlang org doc man gen_server html Function Index addCallback 3 check vsn 0 code change 3 delete callback 3 get callback 2 handle call 3 handle cast 2 handle info 2 init l start_link l stop 0 terminate 2 Function Details addCallback State term Fuseld string Pid pid gt NewState NewState list Adds callback to fuse check vsn Result Result term Checks application version code change OldVsn State term Extra term Result e OldVsn Vsn down Vsn e Result ok NewState term I ferror Reason term Vsn term gen_server code_change 3 11 3 Cluster elements 111 VeilFS Documentation Release 1 0 delete callback State term Fuseld string Pid pid Result e NewState term e Result NewState fuse not found fuse delet
93. h 2 strip_path_leaf 1 time 0 update meta attr 3 Function Details basename Path string gt string Gives file basename from given path create children list Files list Result Result term Creates list of children logical names on the basis of list with veil_documents that describe children create children list Files list TmpAns list Result e Result term Creates list of children logical names on the basis of list with veil_documents that describe children Veil modules 169 VeilFS Documentation Release 1 0 get parent and name from path Path string ProtocolVersion term e Result tuple Gets parent uuid and file name on the basis of absolute path strip path_leaf Path string gt string Strips file name from path time gt Result integer Returns time in seconds update meta attr File file Attr Value term Result file Attr atime mtime ctime size times Updates file_meta record associated with given file record Attr agument decides which field has to be updated with Value There is one exception to this rule if Attr times Value has to be tuple with fallowing format ATimeValue MTime Value CTime Value or ATime Value MTimeValue If there is no file_meta record associated with given file file_meta will be created and whole function call will be blocking Otherwise the
94. h edu pl wrzeszcz agh edu pl 6 3 Support 85 VeilFS Documentation Release 1 0 86 Chapter 6 About CHAPTER 7 Development This document intends to provide essential information for VeilCluster developers General information about VeilCluster usage can be found in the About file 7 1 Repo layout c_src TODO uzupelnic cacerts TODO uzupelnic config TODO uzupelnic include TODO uzupelnic releases folder for generated VeilCluster releases e src cluster elements includes directories that contain code of Erlang modules that enable cluster management and host proper modules of VeilFS veil modules includes directories that contain code of proper modules of VeilFS proto definitions of protocol buffer messages used by clients during the communication with VeilCluster test test_distributed TODO uzupelnic czy to nie moze byc w test e othe VeilCluster files 7 2 Src Directly in the src directory only files needed to start application can be put The cluster_elements includes source of Erlang modules which are responsible for load balancing spawning processes for requests etc This allows implementation of proper modules using sequential code 87 VeilFS Documentation Release 1 0 The veil modules includes directories that contain code of proper modules of VeilFS Each proper module will wor
95. he fuse directory contains sources of the user space file system while the helpers directory includes storage helpers used by FUSE 1 3 Prerequisites In order to compile the project you need to have fallowing additional libraries its headers and all its prerequisites in include ld path Also you need cmake 2 8 e fuse e protobuf e ssl crypto boost gt 1 49 e Itdl globus common VeilFS Documentation Release 1 0 globus_oldgaa globus_openssl globus gsi callback globus gsi cert utils globus gsi sysconfig globus proxy ss globus gsi proxy core globus gsi credential globus openssl error Use this command to install the required dependency packages e Debian Ubuntu Dependencies deb packages apt get install libprotobuf dev libfuse dev fuse libboost dev libglobus dev libtool RHEL CentOS Fedora Dependencies rpm packages yum install fuse fuse libs fuse devel protobuf devel openssl devel cmake28 boost devel boost 1 4 Compilation 1 4 1 The fast way If you just need an RPM package you can just type make s rpm If there was no errors you will get list of generated packages rpm or dep 1 4 2 The standard way VeilClient uses cmake as a build tool thus building process is same as for most cmake based projects However you can use Makefile proxy with following interface Note that s flag is optional it s a silent mode which makes output much pret
96. ields from given document or JSON object dao lib Authors Rafal Slota Copyright This software is released under the MIT License Descritpion DAO helper utility functional methods Those can be used in other modules bypassing worker host and gen server Function Index apply 4 apply 5 strip_wrappers l wrap_record l Function Details apply Module module Method atom synch atom asynch atom Args term ProtocolVersion number any error worker not found Same as apply 5 but with default Timeout apply Module module Method atom synch atom fasynch atom Args term ProtocolVersion number Timeout pos_integer gt any ferror worker not found I error timeout Behaves similar to erlang apply 3 but works only with DAO worker Method calls are made through random gen server Method should be tuple synch Method or asynch Method but if its simple atom synch Method is assumed Timeout argument defines how long should this method wait for response strip wrappers VeilDocOrList veil_document veil_document tu ple I tupleQ Strips veil_document wrapper Argument can be either an veil_document or list of veil_document Alternatively arguments can be passed as ok Arg tuple Its con venient because most DAO methods formats return value formatted that way If the ar gument cannot be converted e g error tup
97. ile string gt Result ErrorDetail term ErrorGeneral atom Result ok ErrorGeneral ErrorDetail Creates file operates only on storage First it checks if file exists If not it creates file delete Storage_helper_info record File string Result ErrorDetail term ErrorGeneral atom e Result ok ErrorGeneral ErrorDetail Deletes file operates only on storage First it checks if file exists and is regular file If everything is ok it deletes file delete dir Storage_helper_info record Dir string Result ErrorDetail term ErrorGeneral atom e Result ok ErrorGeneral ErrorDetail Deletes dir on storage ls error not implemented yet 176 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Lists files in directory on storage mkdir Storage_helper_info record Dir string Result ErrorDetail term ErrorGeneral atom e Result ok ErrorGeneral ErrorDetail Creates dir on storage mv Storage_helper_info record From string To string Result ErrorDetail term ErrorGeneral atom e Result ok ErrorGeneral ErrorDetail Moves file on storage read Storage_helper_info record File string Offset integer Size inte ger gt Result Bytes binary ErrorDetail term ErrorGeneral atom e Result ok Bytes ErrorGeneral Er
98. is request returns list of names of contained files and subdirectories Parameters epath string path to file or directory Response Headers Content Type application json for path to directory Content Type MIME type for path to file Status Codes 200 OK 404 Not Found 500 Internal Server Error Example request GET rest latest files dirl dir2 HTTP 1 1 Header content type application json Host example com Example responses HTTP 1 1 200 OK connection close server Cowboy date Sun 05 Jan 2014 16 34 54 GMT content length 12 Access Control Allow Origin content type application json file txt POST rest latest files path Upload data to specified path using multipart method Path has to be a valid file system path that is it can t contain regular file as a subdirectory Specified path can t exist and will be created automatically 9 4 Resources 97 VeilFS Documentation Release 1 0 Parameters path string path where file will be uploaded Request Headers Content Type multipart form data Response Headers Content Type application octet stream Status Codes 100 Continue 204 No Content 422 Unprocessable Entity An example curl http curl haxx se request to upload file file txt that is located in current directory to remote location dir file txt would be curl i k cert proxy_cert X POST H content type mul
99. iveAtom const google protobuf Message amp fMsg Sends given protobuf message and receives atom 18 Chapter 2 Classes VeilFS Documentation Release 1 0 This method is simalar to FslogicProxy sendFuseReceiveAnswer But receives simple atom cluster response See FslogicProxy sendFuseReceiveAnswer Protected Attributes boost shared ptr lt MessageBuilder gt m messageBuilder MessageBuilder used to construct cluster packets 2 4 veil client ISchedulable class veil client ISchedulable The ISchedulable interface Deriving from this interface gives possibility to schedule and run async delayed tasks using JobScheduler See JobScheduler addTask Public Type TaskID enum The TaskID enum Values e TASK_CLEAR_FILE_ATTR TASK SEND FILE NOT USED TASK RENEW LOCATION MAPPING TASK REMOVE EXPIRI E D_LOCATON_MAPP ING e TASK_PING_CLUSTER TASK_ASYN OCATION J Q Gl H tj H zJ TASK ASYN EADDIR C C_ TASK_ASYNC_GETATTR C_ TASK_ASYNC_UPDATE_TIMES TASK_CLEAR_ATTR TASK CONNECTION HANDSHAKE e TASK LAST ID Public Functions ISchedulable ISchedulable 2 4 veil client ISchedulable 19 VeilFS Documentation Release 1 0 Interface destructor bool runTask TaskID taskId std string argO std string arg1 std s
100. k Public Static Functions 3 11 veilfs h 63 VeilFS Documentation Release 1 0 boost shared_ptr lt JobScheduler gt getScheduler TaskID taskId TASK_LAST_ID Returns JobScheduler assigned to this object boost shared_ptr lt Config gt getConfig Returns Config assigned to this object boost shared_ptr lt SimpleConnectionPool gt getConnectionPool boost shared_ptr lt PushListener gt getPushListener void addScheduler boost shared_ptr lt JobScheduler gt injected Sets JobScheduler object void setConfig boost shared_ptr lt Config gt injected Sets Config object void setConnectionPool boost shared_ptr lt SimpleConnectionPool gt injected void staticDestroy Private Members std string m_root Filesystem root directory uid_t m_uid Filesystem owner s effective uid gid tm gid 64 Chapter 3 Files VeilFS Documentation Release 1 0 Filesystem owner s effective gid uid tm ruid Filesystem root real uid gid tm rgid Filesystem root real gid boost shared_ptr lt FslogicProxy gt m fslogic FslogicProxy instance boost shared_ptr lt StorageMapper gt m storageMapper StorageMapper instance boost shared_ptr lt MetaCache gt m metaCache MetaCache instance boost shared_ptr lt helpers StorageHelperFactory gt m_shFactory Storage Helpers Factory instance std map lt std string std pair lt std string timet gt gt m_linkCache
101. k Copyright This software is released under the MIT License Descritpion This module is responsible for creating dns response for given dns re quest Currently only supported are queries of type a and class in Function Index e generate_answer 5 Function Details generate answer Packet Dispatcher DispatcherTimeout ResponseTTL Protocol Result Dispatcher term DispatcherTimeout non_neg_integer e Packet binary Protocol udp tcp ResponseTTL non_neg_integer e Result ok binary0 I ferror term0 Generates binary dns response for given binary dns request non protocol agnostic dns_worker Authors Bartosz Polnik Copyright This software is released under the MIT License 11 4 Veil modules 165 VeilFS Documentation Release 1 0 Descritpion This module implements worker plugin behaviour to provide func tionality of resolution ipv4 addresses for given worker name Behaviours worker_plugin_behaviour Function Index cleanup 0 env_dependencies 0 handle 2 init l start_listening 0 Function Details cleanup Result e Result ok worker plugin behaviour callback cleanup 0 handle ProtocolVersion term Request Result Error term e Request ping get version update_state list listO get worker atom get_nodes Response inet ip4_address e Result ok fok Response I error Error pong Version Version term worker plugi
102. k inside of worker host one of cluster elements so it must implement worker_plugin_behaviour defined in worker_plugin_behaviour erl file in this directory 7 3 Tests It should contain the same subdirectories as src Each test name should be constructed as follows name of tested file tests erl e g nodemanager_tests erl should contain functions that test code from node manager er file Eunit is used during tests so each test file should include eunit include eunit hrl use proper functions names name of each test function must end with test e use compilation control macros code between ifdef TEST and endif 7 4 Releases To create new release version must be changed in both src veil_cluster_node app src and re leases reltool conf Node Documentation is generated automatically using edoc so it should use tags defined by it 7 5 Development using Makefile to generate single releases and test environments of veil cluster nodes 7 5 1 Generating and managing a single node release for development purposes The script gen_dev generates a single node release for testing purposes It uses veil cluster script to set the configuration Every node worker or CCM requires information about all CCMs running in the cluster Hence to generate release of a node it is required to specify the following set of arguments name node n
103. le is passed this method returns it un changed This method is designed for use as wrapper for get like DAO methods E g 11 4 Veil modules 155 VeilFS Documentation Release 1 0 dao_lib strip_wrappers dao_vfs get_file fabsolute_path foo bar See dao save record I and dao get_record l for more details about veil_document wrapper wrap_record Record tuple gt veil_document Wraps given erlang record with veil_document wrapper See dao save_record l and dao get record I for more details about veil_document wrapper dao share Authors Michat Wrzeszcz Copyright This software is released under the MIT License Descritpion This module provides high level DB API for handling file sharing Function Index e get file share l remove file share l save file share l Function Details get file share Key file File uuid fuser User uuid fuuid UUID uuid gt f ok file share doc fok file share doc error any no return Gets info about file sharing from db by share id file name or user uuid 1 Non error re turn value is always ok ftveil document record share_desc See dao save_record I and dao get record I for more details about Hveil document wrapper Should not be used di rectly use dao handle 2 instead See dao handle 2 for more details remove file share Key file File uuid user User uuid fuuid UUID
104. load view def 2 remove record l save record l set db l Function Details cleanup Result Error timeout I term e Result ok ferror Error worker plugin behaviour callback cleanup 0 doc to term Field term gt term Converts given valid BigCouch document body into erlang term If document contains saved record which is a valid record see is valid record 1 then structure of the returned record will be updated get record Id atom string gt fok veil_document record tuple error Error term no_return Retrieves record with UUID Id from DB Returns whole veil_document record con taining UUID Revision Info and demanded record inside veil_document uuid and veil_document rev_info should not be ever changed You can strip wrappers if you do not need them using API functions of dao_lib module See veil_document structure for more info Should not be used directly use dao handle 2 instead handle ProtocolVersion term Request Result e Args list Error term e Method atom e Request Method Args Mod atom Method Args ping get version Response term e Result ok fok Response I error Error pong Version Version term worker plugin behaviour callback handle 1 All Module Method Args requests second argument executes Method with Args in Otype type dao Module module but with one exception If Module
105. lperFactory getStorageHelper std vector lt std string gt storageHelperArgs Arguments for storage helper See StorageHelperFactory getStorageHelper 5 4 veil client storagelnfo 77 VeilFS Documentation Release 1 0 78 Chapter 5 Structs Part II Veil Cluster 79 VeilFS Documentation Release 1 0 Documents 81 VeilFS Documentation Release 1 0 82 CHAPTER 6 About VeilCluster is a part of VeilFS system that unifies access to files stored at heterogeneous data storage systems that belong to geographically distributed organizations 6 1 Goals The main goal of VeilCluster is to provision a self scalable cluster which manages the VeilFS system in a single data centre i e it stores meta data about actual users data from the data centre decides how to distribute users files among available storage systems and executes data management rules which can be defined by administrators or users 6 2 Getting Started This is a short tutorial how to start VeilCluster on a single machine 6 2 1 Prerequisites In order to compile the project you need to have the following libraries libglobus gsi callback e libglobus common e libssl e libfuse e libboost filesystem thread random system version gt 1 49 Use the following command to install the required dependency packages e RHEL CentOS Fedora Dependencies rpm packages yum install globus proxy utils globus gsi
106. lt Config gt cnf boost shared_ptr lt JobScheduler gt scheduler boost shared_ptr lt FslogicProxy gt fslogic boost shared_ptr lt MetaCache gt metaCache boost shared_ptr lt StorageMapper gt mapper boost shared_ptr lt helpers StorageHelperFactory gt sh factory VeilFS constructor VeilFS int access const char path int mask access FUSE callback Not implemented yet int getattr const char path struct stat statbuf bool fuse_ctx true getattr FUSE callback See int readlink const char path char link size_t size readlink FUSE callback Not implemented yet See int mknod const char path mode_t mode dev_t dev 30 Chapter 2 Classes VeilFS Documentation Release 1 0 mknod FUSE callback See int mkdir const char path mode_t mode mkdir FUSE callback See int unlink const char path unlink FUSE callback See int rmdir const char path rmdir FUSE callback See int symlink const char path const char link symlink FUSE callback Not implemented yet See int rename const char path const char newpath rename FUSE callback See int link const char path const char newpath link FUSE callback Not implemented yet See int chmod const char path mode_t mode chmod FUSE callback 2 12 veil client VeilFS 31 VeilFS Documentation Release 1 0 See int chown const char path uid_t
107. ly if it wasn t released manually void changeType LockType type Changle lock type Releases current lock and locks for diffrent operation type void release Release current lock void lock 11 VeilFS Documentation Release 1 0 Locks current lock Private Members ReadWriteLock amp m lock The wrapped lock LockType m type Type of this lock See LockType bool m released Flag saying if lock is released right now 2 2 veil client Config class veil client Config The Config Parses config files and provides safe access to configuration map Public Functions std string getFuseID Returns current FuselD void negotiateFuseID time_t delay 0 Starts FuseID negotiation process Parameters delay Since this is async actions you can specify execution delay in seconds std string getString std string opt Returns string value of requested option Before using this function you should check is option exists but it s not required See Config isSet int getInt std string opt 12 Chapter 2 Classes VeilFS Documentation Release 1 0 Returns int value of requested option Before using this function you should check is option exists but it s not required See Config isSet double getDouble std string opt Returns double value of requested option Before using this function you should check is option exists but it s not required See
108. me string ViewName string Quer yArgs Hview query argsf ok QueryResult term I error term Execute a given view There are many additional query args that can be passed to a view see query args for details revision Revinfo term gt term Normalize revision info dao_hosts Authors Rafal Slota Copyright This software is released under the MIT License Descritpion This module manages hosts and connections to VeilFS DB Function Index ban l ban 2 call 2 call 3 delete l 152 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 insert l reactivate l store exec 2 Function Details ban Host atom ok error timeout Bans db host name lowers its priority while selecting host in get host 0 Ban will be cleaned after DEFAULT BAN TIME ms or while store refresh ban Host atom BanTime integer ok I ferror no host I error timeout Bans db host name lowers its priority while selecting host in get host 0 Ban will be cleaned after BanTime ms or while store refresh If given Host is already banned nothing happens and will return ok If given Host wasn t inserted returns error no host call Method atom Args term term I error rpc retry limit exceeded Calls fabric Method with Args on random db host Random host will be assigned to the calling process and will be used with subsequent calls call Mo
109. method call will be asyn chronous Returns given as argument file record unchanged unless file_meta had to be created In this case returned file record will have file meta_doc field updated and shall be saved to DB after this call logical_files_manager Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module provides high level file system operations that use logical names of files Function Index change_file_perm 2 chown 0 create l create share 2 create standard share l delete l doUploadTest 4 error_to_string l exists l get ets name 0 get file by uuid I get file full name by uuid I get file name by uuid 1 170 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 get_file_user_dependent_name_by_uuid 1 get share l getfileattr l getfilelocation 1 Is 3 mkdir 1 mv 2 read 3 remove share l rmdir l truncate 2 write 2 write 3 write_from_stream 2 Function Details change file perm FileName string NewPerms integer Result ErrorDetail term ErrorGeneral atom Result ok ErrorGeneral ErrorDetail Changes file s permissions in db chown error not implemented yet Changes owner of file in db create File string gt Result ErrorDetail term ErrorGeneral atom Result ok ErrorGeneral ErrorDetail Creates file uses logical name of file First it c
110. n Req term Cowboy s webscoket_init callback Initialize connection proceed with TLS GSI authentica tion If GSI validation fails connection will be closed Currently validation is handled by Globus NIF library loaded on erlang slave nodes websocket_terminate Reason term Req State ok e Req term e State hander_state Cowboy s webscoket_info callback Connection was closed For more information please refer Cowboy s user manual 11 4 Veil modules Documents 11 4 1 cluster_rengine Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module implements worker_plugin_behaviour to provide the func tionality of rule engine which triggers admin user defined rules when events ap pears Behaviours worker_plugin_behaviour 122 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Function Index e cleanup 0 handle 2 e init l Function Details 11 4 2 gateway Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module implements worker_plugin_behaviour to provide gateway functionality transfer of files between data centers Behaviours worker_plugin_behaviour Function Index e cleanup 0 e handle 2 init l Function Details 11 4 3 remote files manager Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module implem
111. n behaviour callback handle 1 Calling handle _ ping returns pong Calling handle _ get version returns current version of application Calling handle _ update state _ updates plugin state Calling handle _ get worker Name returns list of ipv4 addresses of workers with specified name init Args term Result e Error term e Result dns worker statef error Error worker plugin behaviour callback init 1 start listening gt ok Starts dns listeners and terminates dns worker process in case of error 11 4 11 Fslogic Documents 166 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 fslogic Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module implements worker plugin behaviour to provide func tionality of file system logic Behaviours worker_plugin_behaviour Function Index cleanup 0 create dirs 4 get file 3 get files number 3 get full file name l get user id 0 get user root l handle 2 init l Function Details cleanup ok worker plugin behaviour callback cleanup 0 create dirs Count integer CountingBase integer SHInfo term TmpAns string gt string Creates dir at storage for files if needed Returns the path that contains created dirs get file ProtocolVersion term File string FuselD string Result Result term Gets file info from DB get_
112. n critical section right now int m writers How many writers are waitning in queue to access critical section int m fairness How many readers was in critical section since last writer left class AutoLock The AutoLock class Object of this class provides auto unlocking ReadWriteLock Public Functions AutoLock ReadWriteLock amp lock LockType type 3 9 lock h 55 VeilFS Documentation Release 1 0 Create auto ReadWriteLock Also locks for read write depending on LockType param AutoLock Unlocks wrapped ReadWriteLock lock but only if it wasn t released manually void changeType LockType type Changle lock type Releases current lock and locks for diffrent operation type void release Release current lock void lock Locks current lock Private Members ReadWriteLock amp m_lock The wrapped lock LockType m_type Type of this lock See LockType bool m_released Flag saying if lock is released right now 3 10 messageBuilder h Author Beata Skiba Copyright C 2013 ACK CYFRONET AGH This software is released under the MIT Licence Defines 56 Chapter 3 Files VeilFS Documentation Release 1 0 FUSE_MESSAGES COMMUNICATION_PROTOCOL GET_FILE_LOCATION FUSE_MESSAGE FSLOGIC namespace veil namespace client class MessageBuilder The MessageBuilder class This class can be used to build protobuf messages used to communicate with cluster Theres encode and decode m
113. ne parameter names are preceded by a hyphen 7 6 3 Starting a release There are three options which are used to start a release They are passed along with other arguments to the veil cluster script start gt the script will perform the configuration and then start the node as a daemon attach gt the script will skip configuration and try to attach to a running node with an erla consol gt the script will perform the configuration and then start the node with an erlang sh If none of these arguments occur the script will terminate after setting up the configuration 7 6 4 Full example of usage veil cluster name mynode host net main ccm ccmnode host net console Above command will configure the release according to config args except for name and main ccm which will be modified corresponding to command line arguments Then the node will be started pre senting to the user with an erlang shell 7 7 Support For more information visit project Confluence or write to wrzeszcz agh edu pl wrzeszcz agh edu pl 92 Chapter 7 Development CHAPTER 8 Installation Using current RPM version it is possible to install application s nodes Bigcouch database and config ure storage Undermentioned instructions are valid for all Red Hat Linux distributions especially for Scientific Linux 8 1 Dependencies yum install js devel libicu libicu devel openssl openssl devel python
114. nfo record that shall contain open flags Same record will be returnd with fd field set file descriptor therefore record returned by open shall be passed to next read write release calls read _sh_name string _sh_args string _path string _size integer offset integer fi st fuse file infof ErrorCode integer Data binary I error NIF_not_loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode return value equals to bytes read count if operation was succesfull otherwise negated POSIX error code will be returned read 6 reads _size bytes starting with _offset from given file If the fi arguemnt is given with valid file descriptor fd field the fd will be used to access file Otherwise read 6 will open file for you release _sh_name string _sh_args string path string _fi st_fuse_file_info ErrorCode integer error NIF_not_loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode re 180 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 turn value shall be 0 if operation was succesfull otherwise negated POSIX error code will be returned release 4 clos
115. nformation about storage helper and file id at helper Next it uses storage helper to write data to file write from stream File string Buf binary Result BytesWritten integer ErrorDetail term ErrorGeneral atom e Result BytesWritten ErrorGeneral ErrorDetail Appends data to the end of file uses logical name of file First it gets information about storage helper and file id at helper Next it uses storage helper to write data to file storage_files_manager Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module provides high level file system operations that operates directly on storage Function Index chmod 3 chown 4 create 2 delete 2 delete_dir 2 11 4 Veil modules 175 VeilFS Documentation Release 1 0 e 1s 0 mkdir 2 mv 3 read 4 truncate 3 write 3 write 4 Function Details chmod Storage helper info record Dir string Mode integer Result ErrorDetail term ErrorGeneral atom e Result ok ErrorGeneral ErrorDetail Change file mode at storage chown Storage_helper_info record Dir string User string Group string Result ErrorDetail term ErrorGeneral atom e Result ok ErrorGeneral ErrorDetail 6699 Change file s owner if user or group shouldn t be changed use as an argument create Storage helper info record F
116. ng e User user doc Convinience function to get user name from tveil document encapsulating user record get team names UserQuery term gt string no return Returns list of group team names for given user UserQuery shall be either user record or query compatible with user logic get user l The method assumes that user exists therefore will fail with exception when it doesnt get teams User gt Result e Result string e User user doc Convinience function to get user teams from ftveil document encapsulating user record get user Key login Login string email Email string fuuid UUID user fdn DN string rdnSequence AttributeTypeAnd Value ok user docQ error anyQ noindex Retrieves user from DB by login email uuid DN or rdnSequence proplist Returns veil document wrapping a Huser record invert dn string string gt string Inverts the sequence of entries in a DN string oid code to shortname ferm gt string no return 162 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Converts erlang like OID code to OpenSSL short name rdn sequence to dn string AttributeTypeAndValue gt string no return Converts rdnSequence to DN string so that it can be compared to another DN remove user Key login Login string email Email string uuid UUID user
117. ng node addition to existing cluster it is only required to give IP address of any working node e g 172 16 67 143 8 4 Useful commands opt veil nodes ccm bin veil cluster node attach connect to local CCM Warning to termir opt veil nodes worker bin veil cluster node attach connect to local worker Warning to yum remove veil clean deletion of RPM 8 5 Vocabulary database node installed Bigcouch instance veil node installed CCM and worker or only worker db cluster connected and cooperating database nodes group veil cluster cluster veil connected and cooperating veil nodes group e storage it is composed of group name that use it and mount point group name fuse client group name that use storage storage directory storage mount point Submodules 94 Chapter 8 Installation CHAPTER 9 REST API Documents 9 1 Authentication In brief authentication proceed through following steps Client opens HTTPS connection and sends his certificate in most cases proxy certificate and all other certificates required to associate his certificate with certificate authorities present on cluster Each HTTPS connection is accepted regardless of sent certificates However all certifi cates are saved in connection session If connection requires authentication and in case of REST requests it does immediately after connection is established sent certificates
118. ns their correspondent cluster fslogic meth ods Each object of FslogicProxy has its own connection pool so in order to always use already opened connections there should be only one instance of this class unless you really need connection isolation between successive cluster requests Public Functions FslogicProxy FslogicProxy bool getFileAttr std string logicName protocol fuse_messages FileAttr amp attr Downloads file attributes from cluster bool getFileLocation std string logicName protocol fuse_messages FileLocation amp location Downloads file location info bool getNewFileLocation std string logicName mode_t mode protocol fuse_messages FileLocation amp location 16 Chapter 2 Classes VeilFS Documentation Release 1 0 Query cluser to create new file in DB and get its real location int renewFileLocation std string logicName Try to renew location validity for given file bool getFileChildren std string dirLogicName uint32_t children_num uint32_t offset std vector lt std string gt amp childrenNames List files in given folder std string renameFile std string fromLogicName std string toLogicName Rename move file to new location std string createDir std string logicName mode_t mode Create directory std string deleteFile std string logicName Delete given file bool sendFileNotUsed std string logicName Inform cluster that file isnt used
119. o_return Saves file descriptor to DB Argument should be either file_descriptor record if you want to save it as new document or veil_document that wraps ftfile descriptor if you want to update descriptor in DB See dao save_record l and dao get_record I for more details about ftveil document wrapper Should not be used directly use dao handle 2 instead See dao handle 2 for more details save file File file_info file_doc gt ok uuid error any no_return Saves file to DB Argument should be either file record if you want to save it as new document or veil_document that wraps file if you want to update file in DB See dao save_record l and dao get_record I for more details about Hveil document wrapper Should not be used directly use dao handle 2 instead See dao handle 2 for more details save file meta FMeta file_metaf veil_document gt ok uuid ferror any no_return Saves file_meta to DB Argument should be either Hfile metal record if you want to save it as new document or veil_document that wraps file_meta if you want to update file meta in DB See dao save_record and dao get_record I for more details about veil_document wrapper Should not be used directly use dao handle 2 instead See dao handle 2 for more details save new file FilePath string File file_info gt fok uuid ferror any Ino return Saves new
120. ock m_statMapLock Lock used to synchronize access to MetaCache m_statMap 2 8 veil client PushListener class veil client PushListener Public Functions PushListener PushListener void onMessage const protocol communication_protocol Answer msg Input callback This method should be registered in connection object This is the source of all processed messages int subscribe listener_fun fun Register callback function Each registered by this method function will be called for every in coming PUSH message Registered callback has to return bool value which tells if subscription shall remain active false callback will be removed 24 Chapter 2 Classes VeilFS Documentation Release 1 0 Return ID of subscription that can be used to unsubscribe manually void unsubscribe int subld Remove previously added callback Parameters e subld shall match the ID returned by PushListener subscribe void onChannelError const protocol communication_protocol Answer amp msg Callback called fo each non ok Answer from cluster Protected Functions void mainLoop Worker thread s loop Protected Attributes int m_currentSubld bool m_isRunning boost thread m_worker boost condition m_queueCond boost mutex m_queueMutex boost unordered_map lt int listener_fun gt m_listeners Listeners callbacks std list lt protocol communication_protocol Answer gt m_msgQueue Message in
121. oftware is released under the MIT License Descritpion This module is responsible for handling tcp aspects of dns protocol Behaviours ranch protocol https github com extend ranch blob master manual ranch_protocol md Function Index e loop 5 start link 4 Function Details loop Socket Transport ResponseTTL TCPIdleTime DispatcherTimeout ok Dispatcher Timeout non neg integer e ResponseTTL non neg integer Socket inet socket e TCPIdleTime non_neg_integer Transport term Main handler loop start link Ref term Socket term Transport Result e Pid pidO e Result ok Pid Starts handler dns udp handler Authors Bartosz Polnik term Opts term gt Copyright This software is released under the MIT License Descritpion This module is responsible for handling udp aspects of dns protocol Function Index loop 3 start link 3 164 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Function Details loop Socket ResponseTTL DispatcherTimeout gt no_return DispatcherTimeout non_neg_integer ResponseTTL non_neg_integer Socket inet socket Loop maintaining state start link Port ResponseTTLInSecs DispatcherTimeout ok pidO Dispatcher Timeout non neg integer Port non neg integer ResponseTTLInSecs non neg integer Creates process to handle udp socket dns utils Authors Bartosz Polni
122. oglevel mask 255 as Status terminate Reason State term gt Any term e Reason normal shutdown shutdown term term gen_event terminate 2 11 4 8 Control panel Documents control_panel Authors Lukasz Opiola Copyright This software is released under the MIT License Returns its 11 4 Veil modules 127 VeilFS Documentation Release 1 0 Descritpion This module implements worker_plugin_behaviour callbacks It is re sponsible for setting up cowboy listener and registering nitrogen_handler as the handler Behaviours worker_plugin_behaviour Function Index e cleanup 0 e handle 2 e init l Function Details cleanup Result Error timeout term e Result ok ferror Error worker plugin behaviour callback cleanup 0 Stops cowboy listener and terminates handle ProtocolVersion term Request Result e Error term e Request ping get version Response term Result ok ok Response I error Error pong Version Version term worker plugin behaviour callback handle 1 init Args term Result Error term e Result ok ferror Error worker_plugin_behaviour callback init 1 Sets up cowboy dispatch with nitrogen handler and starts cowboy service on desired port rest_routes Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This module provides mapping of rest subpaths to erlang modules
123. or argp errmsg GLOBUS I GSI PROXY UTILS PRINT ERROR STATUS OK STATUS EXPIRED STATUS NOT FOUND STATUS CANT LOAD STATUS NO NAME STATUS BAD OPTS 68 Chapter 3 Files VeilFS Documentation Release 1 0 STATUS_INTERNAL Functions int globus i gsi proxy utils print error globus result tresult int debug const char filename int line const char fmt int proxy_info int argc char argv FILE _vout FILE _verr This function replaces standard main of grid proxy info tool First two arguments are standard main s arugments The others are used to capture standard output and standard error steams Variables int debug char LONG USAGE 3 17 grid proxy init c Defines SHORT USAGE FORMAT args show short help0 args show full usage args error message errmsg args error argval errmsg args verify next argnum argval errmsg Functions int globus i gsi proxy utils print error globus result tresult int debug const char filename int line const char fmt 3 17 grid proxy init c 69 VeilFS Documentation Release 1 0 int globus i gsi proxy utils pwstdin callback char buf int num int w void globus i gsi proxy utils key gen callback int p int n void dummy int globus I gsi proxy utils extension callback globus gsi callback data t callback data X509 EXTENSION extension int proxy init int argc char argv FI
124. orbidden 404 Not Found 405 Method Not Allowed 9 3 Getting started REST API is based on open standards you can use any web development language or com mand line tool capable of generating an HTTP request to access the API 9 4 Resources REST ish endpoint for interacting with VeilFS 9 4 1 Index rest latest files GET rest latest files path GET POST PUT DELETE rest latest attrs rest latest attrs path GET rest latest shares GET POST rest latest shares guid GET DELETE 9 4 2 rest latest files Methods GET rest latest files Retrieve content of root directory and return as a list of names of files and subdirectories Response Headers Content Type application json Status Codes 200 OK 500 Internal Server Error Example request 96 Chapter 9 REST API VeilFS Documentation Release 1 0 GET rest latest files HTTP 1 1 Header content type application json Host example com Example responses HTTP 1 1 200 OK connection close server Cowboy date Sun 05 Jan 2014 16 34 54 GMT content length 12 Access Control Allow Origin content type application json arri dira Groups Erle CXE 9 4 3 rest latest files path Methods GET rest latest files path Retrieve content of specified file or directory For path to an existing file this request returns its content For path to an existing directory th
125. ose two arguments decide which Storage Helper shall be used for this operation ErrorCode re turn value shall be 0 if operation was succesfull otherwise negated POSIX error code will be returned chown 5 changes file s uid and gid 11 4 Veil modules 179 VeilFS Documentation Release 1 0 fsync _sh_name string sh args string path string isdatasync integer fi st_fuse_file_info ErrorCode integer error NIF not loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode re turn value shall be 0 if operation was succesfull otherwise negated POSIX error code will be returned getattr sh name string sh args string path string gt ErrorCode integer Stats st_stat I ferror NIF not loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode return value shall be 0 if operation was succesfull otherwise negated POSIX error code will be re turned gettattr 3 returns st_stat record for given file path Note that if ErrorCode does not equal 0 fields of st_stat are undefined and shall be ignored mkdir _sh_name string sh args string _path string mode inte ger ErrorCode integer
126. per shall be used for this operation ErrorCode return value shall be 0 if operation was succesfull otherwise negated POSIX error code will be re turned statfs 3 returns st_statvfs record for given path See statfs syscall for more details truncate _sh_name string sh args string _path string size inte ger ErrorCode integer error NIF not loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode re turn value shall be 0 if operation was succesfull otherwise negated POSIX error code will be returned truncate 4 changes file size to size unlink _sh_name string _sh_args string _path string ExrorCode integer error NIF_not_loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode re turn value shall be 0 if operation was succesfull otherwise negated POSIX error code will be returned unlink 3 removes given file not directory write _sh_name string _sh_args string _path string _buf binary offset integer fi st_fuse_file_info ErrorCode integer error NIF not loaded First 2 arguments of this method should come from storage_helper_info record Those two argumen
127. plication http www erlang org doc man application html 11 1 1 Function Index start 2 stop l 11 1 2 Function Details start _StartType any _StartArgs any gt Result e Error already started pid shutdown term termO e Result ok pidQ I ignore error Error Starts application by supervisor initialization stop _State any Result Result ok Stops application 11 2 veil cluster node sup Authors Michal Wrzeszcz Copyright This software is released under the MIT License 107 VeilFS Documentation Release 1 0 Descritpion It is the main supervisor It starts as it child node manager which initializes node Behaviours supervisor http www erlang org doc man supervisor html 11 2 1 Function Index e init l start link l 11 2 2 Function Details init Args term Result Modules module dynamic RestartPolicy permanent I transient temporary Result ok SupervisionPolicy ChildSpec ignore StartFunc M module F atom A term undefined pos_integer supervisor init 1 start link Args term gt Result e Error already started pid shutdown term termO e Result ok pidQ I ignore error Error Starts application supervisor Submodules 11 3 Cluster elements Documents 11 3 1 cluster_manager Authors Michal Wrzeszcz Copyright This software is released under the MIT Licens
128. processable requests No need to check if file exists as the same will be done in post method Utils Documents openid_utils Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This is a simple library used to establish an OpenID authentication It needs nitrogen and ibrowse to run Function Index e get login url 2 nitrogen prepare validation parameters 0 e nitrogen retrieve user info 0 e validate_openid_login l Function Details get login url HostName string RedirectParams string Result Result ok error endpoint unavailable Produces an URL with proper GET parameters used to redirect the user to OpenID Provider login page RedirectParams are parameters concatenated to return to field nitrogen prepare validation parameters gt Result e Error invalid request Result string string I error Error 138 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 This function retrieves endpoint URL and parameters from redirection URL created by OpenID provider They are later used as arguments to validate_openid_login function Must be called from within nitrogen page context to work precisely from openid redirection page nitrogen retrieve user info Result e Error invalid request e Result ok list I error Error This function retrieves user info from parameters of redirection URL created by OpenID provider They ar
129. proxy_certificate OtpCert OTPCertificate gt boolean Checks is given OTP Certificate has an proxy extension 116 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 load_certs CADir string gt ok no_return Loads all PEM encoded CA certificates from given directory along with their CRL certificates if any Note that CRL certificates should also be PEM encoded and the CRL filename should match their CA filename but with crl extension proxy subject OtpCert OTPCertificate gt rdnSequence AttributeTy peAndValue Returns subject of given certificate If proxy certificate is given EEC subject is returned update_crls CADir string gt ok no_return Updates CRL certificates based on their distribution point x509 CA extension Not yet fully implemented verify_callback OtpCert OTPCertificate Status term Certs 4 OT PCertificate valid UserState any I fail Reason term This method is an registered callback called foreach peer certificate This callback saves whole certificate chain in GSI ETS based state for further use gsi_nif Authors Rafal Slota Copyright This software is released under the MIT License Descritpion This module loads GSI NIF libs Function Index start l verify_cert_c 4 Function Details start Prefix string ok ferror Reason term Thi
130. r Should not be used directly use dao handle 2 instead See dao handle 2 for more details remove user Key login Login string femail Email string fuuid UUID uuid dn DN string gt error any no_return Removes user from DB by login e mail uuid or dn Should not be used directly use dao handle 2 instead See dao handle 2 for more details save user User user_info user_doc gt ok user ferror any no_return Saves user to DB Argument should be either user record if you want to save it as new document or veil_document that wraps user if you want to update descriptor in DB See dao save_record l and dao get_record I for more details about veil_document wrapper Should not be used directly use dao handle 2 instead See dao handle 2 for more details dao vfs Authors Rafal Slota Copyright This software is released under the MIT License Descritpion This module gives high level DB API which contain veil file system specific methods All DAO API functions should not be called directly Call dao handle _ vfs MethodName ListOfArgs instead See dao handle 2 for more details Function Index count_subdirs 1 find_files l get_descriptor l get file l get file metarl get path inforl 11 4 Veil modules 157 VeilFS Documentation Release 1 0 get_storage l list_descriptors 3 list_dir 3 list_storage O lock_file 3
131. r database does not exist error term Deletes db named DbName delete db DbName string Opts Option ok error database does not exist ferror term e Option atom atom term0 Deletes db named DbName delete doc DbName string DocID string ok I error missing I error deleted I error term Deletes doc from db delete_docs DbName string DocID string ok error term Deletes list of docs from db gen_uuid gt string Generates UUID with CouchDBs utc_random algorithm get_db_info DbName string ok instance start time binary doc_count non_neg_integer doc_del_count non_neg_integer purge_seq non_neg_integer compact running boolean disk size non_neg_integer disk format version pos integer I ferror database does not exist I error term Returns db info for the given DbName get doc count DbName string ok non neg integerQ error database does not exist I error term Returns doc count for given DbName insert doc DbName string Doc doc ok RevNum non neg integer RevBin binaryQ I error conflict error term Inserts doc to db insert_doc DbName string Doc doc Opts Option ok RevNum non neg integer RevBin binary ferror conflict error term e Option atom atom term
132. r at once Note that each opendir syscall will query at least DIR_BATCH_SIZE dirents 58 Chapter 3 Files VeilFS Documentation Release 1 0 namespace veil namespace client class VeilFS The VeilFS main class This class contains FUSE all callbacks so it basically is an heart of the filesystem Technically VeilF S is an singleton created on programm start and registred in FUSE daemon Public Functions VeilFS std string path boost shared_ptr lt Config gt cnf boost shared_ptr lt JobScheduler gt scheduler boost shared ptr lt FslogicProxy gt fslogic boost shared ptr lt MetaCache gt metaCache boost shared ptr lt StorageMapper gt mapper boost shared ptr lt helpers StorageHelperFactory gt sh factory VeilFS constructor VeilFS int access const char path int mask access FUSE callback Not implemented yet int getattr const char path struct stat statbuf bool fuse_ctx true getattr FUSE callback See int readlink const char path char link size_t size readlink FUSE callback Not implemented yet See int mknod const char path mode_t mode dev_t dev mknod FUSE callback See 3 11 veilfs h 59 VeilFS Documentation Release 1 0 int mkdir const char path mode_t mode mkdir FUSE callback See int unlink const char path unlink FUSE callback See int rmdir const char path rmdir FUSE callback See int symlink
133. reates file in db and gets information about storage helper and file id at helper Next it uses storage helper to create file on storage create_share File string Share_With term Result ErrorDetail term ErrorGeneral atom e Result ok Share info ErrorGeneral ErrorDetail Share info term Creates share info for file file path is an argument create standard share File string gt Result Veil modules 171 VeilFS Documentation Release 1 0 ErrorDetail term ErrorGeneral atom e Result ok Share info ErrorGeneral ErrorDetail Share info term Creates standard share info share with all for file file path is an argument delete File string Result ErrorDetail term ErrorGeneral atom e Result ok ErrorGeneral ErrorDetail Deletes file uses logical name of file First it gets information about storage helper and file id at helper Next it uses storage helper to delete file from storage Afterwards it deletes information about file from db doUploadTest File string WriteFunNum integer Size integer Times integer Result e BytesWritten integer e Result BytesWritten WriteTime e WriteTime integer Tests upload speed error to string Error term Result e Result string Translates error to text message exists File string Result ErrorDetail term ErrorGeneral
134. ring I oid code to shortnamerl e rdn sequence to dn string I remove user l shortname to oid coderl e sign in l update dn list 2 update email list 2 update teams 2 Function Details create user Login Name Teams Email DnList Result DnList string e Email string Login string e Name string e Result ok user docQ I error anyQ Teams string Creates a user in the DB extract dn from cert PemBin binary rdnSequence AttributeType AndValue 1 ferror Reason 11 4 Veil modules 161 VeilFS Documentation Release 1 0 Reason proxy ceertificate self signed extraction_failed Processes a pem certificate and extracts subject DN part returning it as an rdnSequence Returns an error if fails to extract DN certificate is a proxy certificate gt error proxy_ceertificate certificate is self signed gt error self signed get dn list User Result e Result string e User user doc Convinience function to get DN list from ftveil document encapsulating user record get email list User gt Result e Result string e User user doc Convinience function to get e mail list from ftveil document encapsulating user record get login User Result e Result string e User user doc Convinience function to get user login from ftveil document encapsulating user record get name User gt Result e Result stri
135. rorDetail Reads file operates only on storage First it checks file attributes file type and file size If everything is ok it reads data from file truncate Storage helper info record File string Size integer gt Result ErrorDetail term ErrorGeneral atom Result ok ErrorGeneral ErrorDetail Truncates file operates only on storage First it checks if file exists and is regular file If everything is ok it truncates file write Storage_helper_info record File string Buf binary Result BytesWritten integer ErrorDetail term ErrorGeneral atom e Result BytesWritten ErrorGeneral ErrorDetail Appends data to the end of file operates only on storage First it checks file attributes file type and file size If everything is ok it reads data from file write Storage_helper_info record File string Offset integer Buf bi nary Result BytesWritten integer ErrorDetail term ErrorGeneral atom e Result BytesWritten ErrorGeneral ErrorDetail 11 4 Veil modules 177 VeilFS Documentation Release 1 0 Writes data to file operates only on storage First it checks file attributes file type and file size If everything is ok it reads data from file veilhelpers Authors Rafal Slota Copyright This software is released under the MIT License Descritpion This module shoul be used as an pro
136. rts the server stop ok Stops the server terminate Reason State term gt Any term Reason normal shutdown shutdown term term gen_server terminate 2 ws_handler Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module forwards requests from socket to dispatcher Function Index init 3 websocket_handle 3 websocket_info 3 websocket_init 3 websocket_terminate 3 11 3 Cluster elements 121 VeilFS Documentation Release 1 0 Function Details init Proto term Req term Opts term upgrade protocol cow boy websocket Switches protocol to WebSocket websocket handle Type atom Data term Req State reply Type atom Data term Req State ok Req State I shut down Req State e Req term State hander_state Cowboy s webscoket_handle callback Binary data was received on socket For more infor mation please refer Cowboy s user manual websocket_info Msg term Req State reply Type atom Data term Req State ok Req State shutdown Req State Req term State hander_state Cowboy s webscoket_info callback Erlang message received For more information please refer Cowboy s user manual websocket_init TransportName atom Req term Opts list ok Req term State term shutdow
137. s method loads NIF library into erlang VM This should be used once before using any other method in this module verify cert c PeerCert binary PeerChain binary CAChain bi nary CRLs binary ok 1 I ok 0 Errno integer error Reason term This method validates peer certificate All input certificates should be DER encoded ok 1 result means that peer is valid ok 0 Errno means its not Errno is a raw errno returned by openssl globus library error Reason on the other hand is returned only when something went horribly wrong during validation logger Authors Lukasz Opiola Copyright This software is released under the MIT License 11 3 Cluster elements 117 VeilFS Documentation Release 1 0 Descritpion This module handles log dispatching Function Index dispatch_log 5 get console loglevel 0 get current loglevel 0 get default loglevel 0 get include stacktrace O loglevel atom to int I loglevel int to atom I parse process inforl set console loglevel l set include stacktrace l set loglevel I should log l Function Details dispatch log LoglevelAsInt integer Metadata tuple Format string Args string IncludeStacktrace boolean ok I error lager not running Logs the log locally it will be intercepted by central logging backend and sent to central sink get console loglevel gt integer Returns curr
138. start_link Type Result e Error already_started Pid term O e Pid pidO 112 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 e Result ok Pid ignore error Error Type test worker worker ccm Starts the server stop ok Stops the server terminate Reason State term gt Any term e Reason normal shutdown shutdown term term gen_server terminate 2 11 3 3 worker_host Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion This module hosts all VeilFS modules fslogic cluster_rengin etc It makes it easy to manage modules and provides some basic functionality for its plug ins VeilFS modules e g requests management Behaviours gen_server http www erlang org doc man gen_server html Function Index clear_cache l code_change 3 create_simple_cache l create_simple_cache 3 create simple cache 4 create simple cache 5 generate sub proc list I generate sub proc list 5 handle call 3 handle cast 2 handle infor 2 init 1 start link 3 start sub proc S stop l terminate 2 11 3 Cluster elements 113 VeilFS Documentation Release 1 0 Function Details clear cache Cache term ok Clears chosen caches at all nodes code change OldVsn State term Extra term Result e OldVsn Vsn down Vsn e Result ok NewState term I error Reason term
139. stead dao_helper Authors Rafal Slota 11 4 Veil modules 149 VeilFS Documentation Release 1 0 Copyright This software is released under the MIT License Descritpion Low level BigCouch DB API Function Index create db l create db 2 create view 6 delete db l delete db 2 delete doc 2 delete docs 2 gen uuid 0 get db inforl get doc count I insert_doc 2 insert_doc 3 insert_docs 2 insert_docs 3 list_dbs 0 list_dbs l name l open_design_doc 2 open_doc 2 open_doc 3 query_view 3 query view 4 revision l Function Details create db DbName string ok error term Creates db named DbName If db already does nothing and returns ok create db DbName string Opts Option ok error term Option atom atom term Creates db named DbName with Opts Options can include values for q and n for example q 8 and n 3 which control how many shards to split a database into and how many nodes each doc is copied to respectively 150 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 create _view DbName string DesignName string ViewName string Map string Reduce string DesignVersion integer ok I error term 6699 Creates view with given Map and Reduce function When Reduce reduce function won t be created delete db DbName string ok I erro
140. storage_helper_info record which describes storage helper that is connected with given storage described with storage_info record Each storage can have multiple stor age helpers that varies between FUSE groups so that different FUSE clients with different FUSE ID could select different storage helper insert storage HelperName string HelperArgs string gt term 168 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 Creates new mock storage info in DB that uses default storage helper with name HelperName and argument list HelperArgs insert_storage HelperName string HelperArgs string Fuse_groups list gt term Creates new mock storage info in DB that uses default storage helper with name HelperName and argument list HelperArgs TODO This is mock method and should be replaced by GUI tool form control_panel module select storage FuselD string StorageList Hstorage infof stor age_info Chooses and returns one storage_info from given list of storage_info records TODO This method is an mock method that shall be replaced in future Currently returns random stor age_info fslogic_utils Authors Rafal Slota Copyright This software is released under the MIT License Descritpion This module exports utility tools for fslogic Function Index basename l create_children_list 1 create_children_list 2 get_parent_and_name_from_pat
141. string path or uuid UUID remove share Key file File uuid user User uuid fuuid UUID uuid gt Result ErrorDetail term ErrorGeneral atom e Result ok ErrorGeneral ErrorDetail Removes info about share from db rmdir DirName string Result ErrorDetail term ErrorGeneral atom e Result ok ErrorGeneral ErrorDetail Deletes directory in db truncate File string Size integer Result ErrorDetail term ErrorGeneral atom 174 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 e Result ok ErrorGeneral ErrorDetail Truncates file uses logical name of file First it gets information about storage helper and file 1d at helper Next it uses storage helper to truncate file on storage write File string Buf binary Result BytesWritten integer ErrorDetail term ErrorGeneral atom e Result BytesWritten ErrorGeneral ErrorDetail Appends data to the end of file uses logical name of file First it gets information about storage helper and file id at helper Next it uses storage helper to write data to file write File string Offset integer Buf binary Result BytesWritten integer ErrorDetail term ErrorGeneral atom Result BytesWritten ErrorGeneral ErrorDetail Writes data to file uses logical name of file First it gets i
142. string uname Updates file s owner std string changeFileGroup std string path gid_t gid std string gname Updates file s group owner void pingCluster std string nth bool runTask TaskID taskId std string argO std string argl std string arg3 Task runner derived from Schedulable See ISchedulable runTask Protected Functions bool sendFuseReceiveAnswer const google protobuf Message amp fMsg google protobuf Message amp response Sends and receives given protobuf message High level method used to send serialized protobuf message to cluster and return its response as given by reference ob ject Both message and response types have to be subtype of FuseMessage Return true only if received response message is initialized see google protobuff Message IsInitialized std string sendFuseReceiveA tom const google protobuf Message amp fMsg Sends given protobuf message and receives atom This method is simalar to Fslog icProxy sendFuseReceiveAnswer But receives simple atom cluster response See FslogicProxy sendFuseReceiveAnswer 52 Chapier 3 Files VeilFS Documentation Release 1 0 Protected Attributes boost shared_ptr lt MessageBuilder gt m_messageBuilder MessageBuilder used to construct cluster packets 3 8 gsiHandler h Defines GSLINIT COMMAND GSLINFO COMMAND BASE DOMAIN namespace veil namespace client namespace gsi
143. t Proplist id 123 message mess RecordTuple some_record Fields id message The function will produce some_record id 123 11 4 Veil modules 139 VeilFS Documentation Release 1 0 message mess NOTE rest_utils hrl contains convienience macro that will include this function in code with Fields listed for given record name Web gui Documents gui_utils Authors Lukasz Opiola Copyright This software is released under the MIT License Descritpion This file contains useful functions commonly used in control_panel modules Function Index apply or redirect 3 apply or redirect 4 can view logs 0 dn and storage defined 0 empty page 0 get requested hostname 0 get user dn 0 logotype footer I storage defined 0 top menu l top menu 2 user logged in O Function Details apply or redirect Module atom Fun atom boolean gt boolean Checks if the client has right to do the operation is logged in and possibly has a certificate DN defined If so it executes the code apply or redirect Module atom Fun atom Args list boolean gt boolean Checks if the client has right to do the operation is logged in and possibly has a certificate DN defined If so it executes the code can view logs gt boolean Determines if current user is allowed to view cluster logs dn and storage defined gt boolean Convienience function to check both con
144. t get install libglobus gsi callback dev x RHEL CentOS Fedora Dependencies rpm packages yum install globus gsi callback devel Additionally if you want to use NIF API you need to have Erlang NIF libraries and headers Norn 105 VeilFS Documentation Release 1 0 10 1 3 Compilation The easiest way of compiling this lib is to use x rebarx Rebar automatically adds required by Unfortunately you still need to explicate specify _ lglobus _gsi _callback lglobus _common 1ss your _rebar config_ file like in this https github com basho rebar blob master rebar config 10 2 VeilHelpersNIF VeilHelpersNIF is an NIF wrapper for VeilHelpers library 10 2 1 Files x xxvelilhelpers_nif cc NIF wrapper for IStorageHelper interface s VeilHelper source for nm x xxterm_translator cc Helper functions used to translate erlang terms to c c types 10 2 2 Prerequisites All you need are Erlang NIF libraries and headers Normally they are shipped with Erlang 10 2 3 Compilation The wrapper should be compiled as a part of cluster using rebar because rebar automatically 106 Chapter 10 C modules CHAPTER 11 Erlang modules Documents 11 1 veil cluster node app Authors Michal Wrzeszcz Copyright This software is released under the MIT License Descritpion Itis the main module of application It lunches supervisor which then initializes appropriate components of node Behaviours ap
145. tBool std string opt Returns boolean value of requested option Before using this function you should check is option exists but it s not required See Config isSet Warning If given opition wasn t set you ll get empty object of given type T TO bool isSet std string opt Checks if given option is set See Config getValue void setGlobalConfigFile std string path Sets path to global config file See Config parseConfig void setUserConfigFile std string path Sets path to user config file See Config parseConfig 3 6 config h 45 VeilFS Documentation Release 1 0 void setEnv Saves current CWD and HOME env viariables This is re quired as FUSE changes them after non debug start This is also done automatically in Config Config bool parseConfig Parses config from files set by Config setGlobalConfigFile and Config setUserConfigFile User config overides global settings If user config declares all required options global config file isn t required other wise it has exists Config Config Public Static Functions std string absPathRelToC WD std string path Converts relative path to absolute using CWD env as base prefix std string absPathRelToHOME std string path Converts relative path to absolute using HOME env as base prefix void putEnv std string name std string value Saves given env variable Protected Functions template
146. ter 11 Erlang modules VeilFS Documentation Release 1 0 e Reply term e Result reply Reply NewState reply Reply NewState Timeout reply Reply NewState hibernate noreply NewState noreply NewState Timeout noreply NewState hibernate I stop Reason Reply NewState stop Reason NewState Timeout non neg integer infinity gen server handle call 3 handle cast Request term State term Result e NewState term e Result noreply NewState noreply NewState Timeout noreply NewState hiber nate stop Reason term NewState Timeout non_neg_integer infinity gen_server handle_cast 2 handle_info Info timeout term State term Result NewState term e Result noreply NewState noreply NewState Timeout noreply NewState hiber nate stop Reason term NewState Timeout non_neg_integer infinity gen_server handle_info 2 init Args term Result e Result ok State fok State Timeout ok State hibernate stop Reason term ignore State term Timeout non_neg_integer infinity gen_server init l start link Plugin PlugInArgs LoadMemorySize Result e Error already started Pid term O LoadMemorySize integer e Pid pidO e Plugin atom e PlugInArgs any e Result ok Pid ignore error Error Starts host with apropriate plug in start_su
147. that are allowed and directed at specific Id e g if Id undefined gt lt gt lt gt if Id undefined gt lt gt lt gt lt gt content types provided req binary binaryQ reqO Should return list of provided content types without specified ID e g rest resource Should take into account different types of methods PUT GET etc if needed Should return empty list if method is not supported content types provided req binary binary binaryQ reqO Should return list of provided content types with specified ID e g rest resource some_id Should take into account different types of methods PUT GET etc if needed Should return empty list if method is not supported delete req binary binary boolean regO Will be called for DELETE request on given ID Should try to remove specified resource and return true false indicating the result Should always return false if the method is not supported exists req binary binary gt boolean req0 11 4 Veil modules 133 VeilFS Documentation Release 1 0 Should return whether resource specified by given ID exists Will be called for GET PUT and DELETE when ID is contained in the URL get req binary gt term stream integer function halt regO req0 Will be called for GET request without specified ID e g rest resource Sho
148. tier because it leaves only cmakes stdout stderr Configure make s configure Build make s build 8 Chapter 1 About VeilFS Documentation Release 1 0 Install make s install RPM DEB packages make s rpm Testing There are two testing targets make s test which has summarized output per test case and make s cunit which shows detailed test results 1 5 Using VeilClient 1 5 1 Configuration First of all you should tune up some config settings Configuration file can be found in IN STALL PREFIX etc veilFuse conf In most linux distros deafault INSTALL PREFIX is usr local Configuration options are described in configuration file itself In most cases you want to stick with default values although there are 2 options that requires special care cluster hostname hostname of VeilCluster used by client e peer certificate file path to proxy certificate pem file used in SSL session Paths are relative to HOME env unless absolute path is specified You don t edit this global config file if you don t want to You can also create new file type options that shall be overriden and pass config path to your config file option while starting VeilClient Also its possible to override options by setting env variable with the same name only uppercase CLUSTER_HOSTNAME some hostname com veilFuse mount point 1 5 2 Mounting the filesystem Prerequisites In order
149. tipart form data F file fi Example request POST rest latest files dir file txt HTTP 1 1 Host example com Header content type multipart form data Data file file txt Example responses H H P 1 1 100 Continue P 1 1 204 No Content connection close server Cowboy date Fri 24 Jan 2014 08 43 05 GMT content length 0 Access Control Allow Origin x content type application octet stream PUT rest latest files path Upload data to specified path using multipart method Path has to be a valid file system path that is it can t contain regular file as a subdirectory If specified path doesn t exist it will be created automatically If specified path exists it will be overwritten Parameters path string path where file will be uploaded Request Headers Content Type multipart form data Response Headers Content Type application octet stream Status Codes 100 Continue 204 No Content 422 Unprocessable Entity An example curl http curl haxx se request to upload file file txt that is located in current directory to remote location dir file txt would be curl i k cert proxy cert X PUT H content type multipart form data F file fil Example request 98 Chapter 9 REST API VeilFS Documentation Release 1 0 PUT rest latest files dir file txt HTTP 1 1 Host example com Header content type multipart form data Data
150. to use VeilClient you need to have fallowing additional libraries in Id path e libfuse e libprotobuf e libssl Use this command to install the required dependency packages 1 5 Using VeilClient 9 VeilFS Documentation Release 1 0 e Debian Ubuntu Dependencies deb packages apt get install libprotobuf libfuse fuse e RHEL CentOS Fedora Dependencies rpm packages yum install fuse fuse libs protobuf openssl Starting In order to mount VeilFS just enter veilFuse mount point Additionally you can add d option which enables debug mode In debug mode application will remain running displaing all logs and debug informations also in this mode ctrl c unmount filesystem If not in debug mode application will go background as daemon 1 5 3 Unmounting the filesystem If veilFuse was started with d option just hit ctrl c If not fusermount u mount point 1 6 Support For more information visit project Confluence or write to wrzeszcz agh edu pl wrzeszcz agh edu pl Submodules 10 Chapter 1 About CHAPTER 2 Classes Documents 2 1 veil client AutoLock class veil client AutoLock The AutoLock class Object of this class provides auto unlocking ReadWriteLock Public Functions AutoLock ReadWriteLock amp lock LockType type Create auto ReadWriteLock Also locks for read write depending on LockType param AutoLock Unlocks wrapped ReadWriteLock lock but on
151. tring arg3 Callback which are called by JobScheduler when requested See JobScheduler 2 5 veil client JobScheduler class veil client JobScheduler The JobScheduler class Objects of this class are living daemons threads with their own run queue from which they are precessing tasks Public Functions JobScheduler JobScheduler bool hasTask Schedulable TaskID task void add Task Job job Insert register new task to run queue Inserted task shall run when current time passes its Job when See Job void delete Jobs ISchedulable subject ISchedulable TaskID task Deletes all jobs registred by given object Used mainly when ISchedulable object is destructed Protected Functions void schedulerMain 20 Chapter 2 Classes VeilFS Documentation Release 1 0 Thread main loop Checks run queue and runs tasks when needed void runJob Job job Starts given task See JobScheduler schedulerMain void startDaemon Starts restarts daemon Protected Attributes std priority_queue lt Job gt m jobQueue Run queue pthread_t m_daemon Thread ID pthread_mutex_t m_mutex Mutex used to synchronize access to JobScheduler m_jobQueue pthread_cond_t m_queueCond Condition used to synchronize access to JobScheduler m_jobQueue Protected Static Functions void schedulerMainWrapper void arg C wrapper used to start JobScheduler schedulerMain 2 6 veil client MessageBuilder cl
152. ts decide which Storage Helper shall be used for this operation ErrorCode return value equals to bytes writen count if operation was succesfull otherwise negated POSIX error code will be returned write 6 writes _buf binary data to given file starting with _offset _fi argument has the same meaning as in read 6 11 4 Veil modules 181 VeilFS Documentation Release 1 0 182 Chapter 11 Erlang modules Part III MIT License 183 VeilFS Documentation Release 1 0 Copyright C 2014 ACK CYFRONET AGH Permission is hereby granted free of charge to any person obtaining a copy of this software and associated documentation files the Software to deal in the Software without restric tion including without limitation the rights to use copy modify merge publish distribute sublicense and or sell copies of the Software and to permit persons to whom the Software is furnished to do so subject to the following conditions The above copyright notice and this permission notice shall be included in all copies or sub stantial portions of the Software THE SOFTWARE IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND EX PRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGE MENT IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM DAMAGES OR OTHER LIABILITY WHETHER IN AN ACTION OF CONTRACT TORT OR OTHERWISE ARIS
153. ts for this file Parameters e logical_name File path relative to VeilF S mount point useCluster Specify if the method should use cache only deafault or try quering cluster std string findLocation std string logicalName Query cluster about file location and instert it to cache See StorageMapper addLocation void addLocation std string logicalName protocol fuse_messages FileLocation location Cache given file location Insert to file location cache new FileLocation received from cluster 2 10 veil client StorageMapper 27 VeilFS Documentation Release 1 0 void openFile std string logicalName Increases open file count for specified file See locationInfo opened void releaseFile std string logicalName Decreases open file count for specified file See locationInfo opened bool runTask TaskID taskId std string argO std string arg1 std string arg3 Task runner derived from ISchedulable See ISchedulable runTask Protected Attributes std map lt int storagelnfo gt m_storageMapping Contains storage info accessd by its ID See storagelnfo ReadWriteLock m storageMappingLock Lock used while operating on StorageMapper m_storageMapping See StorageMapper m storageMapping std map lt std string locationinfo gt m_file Mapping Contains storage info accessd by its ID See storagelnfo ReadWriteLock m_fileMappingLock Lock used while operationg on StorageMapper m_fileM
154. uid gid_t gid chown FUSE callback Not implemented yet See int truncate const char path off_t newSize truncate FUSE callback See int utime const char path struct utimbuf ubuf utime FUSE callback Not implemented yet See int open const char path struct fuse file info filelnfo open FUSE callback See int read const char path char buf size t size off t offset struct fuse file info filelnfo read FUSE callback See int write const char path const char buf size t size off t offset struct fuse file info filelnfo write FUSE callback See int statfs const char path struct statvfs statInfo 32 Chapter 2 Classes VeilFS Documentation Release 1 0 statfs FUSE callback Not implemented yet See int flush const char path struct fuse file info filelnfo flush FUSE callback Not implemented yet See int release const char path struct fuse file info filelnfo release FUSE callback See int fsyne const char path int datasync struct fuse file info fi fsync FUSE callback Not implemented yet See int setxattr const char path const char name const char value size_t size int flags setxattr FUSE callback Not implemented yet See int getxattr const char path const char name char value size_t size getxattr FUSE callback Not implemented yet See int listxattr const char path char
155. uld return one of the following 1 ResponseBody of the same type as content types provided l re turned for this request 2 Cowboy type stream function serving content of the same type as content_types_provided l returned for this request 3 halt atom if method is not supported get req binary binary gt termO stream integer function halt req req Will be called for GET request with specified ID e g rest resource some_id Should return one of the following 1 ResponseBody of the same type as content_types_provided 2 returned for this request 2 Cowboy type stream function serving content of the same type as content_types_provided 2 returned for this request 3 halt atom if method is not supported post req binary binary term boolean true binary regO Will be called for POST request after the request has been validated Should handle the request and return true false indicating the result Should always return false if the method is not supported Returning true URL will cause the reply to contain 201 redirect to given URL put reg binary binary term boolean reqQ Will be called for PUT request on given ID after the request has been validated Should handle the request and return true false indicating the result Should always return false if the method is not supported validate req binary binary term boolean
156. xy to veilhelpers_nif module This module controls way of accessing veilhelpers_nif methods Function Index exec 2 exec 3 Function Details exec Method atom SHInfo storage_helper_info Arg term D gt error Reason term Response Response term Executes apply veilhelper nif Method Args through slave node Before executing fields from struct SHInfo are preappend to Args list You can also skip SHInfo argument in order to pass exact Args into target Method veilhelpers nif Authors Rafal Slota Copyright This software is released under the MIT License Descritpion This module wraps storage helper s methods using NIF driver Function Index access 4 chmod 4 chown 5 chown name 5 fsync 5 get_flag 3 getattr 3 is_dir 3 is_reg 3 mkdir 4 178 Chapter 11 Erlang modules VeilFS Documentation Release 1 0 mknod 5 open 4 read 5 read 6 release 4 rename 4 rmdir 3 start l statfs 3 truncate 4 unlink 3 write 5 write 6 Function Details access sh name string sh args string path string mask inte ger 3 ErrorCode integer error NIF not loaded First 2 arguments of this method should come from storage_helper_info record Those two arguments decide which Storage Helper shall be used for this operation ErrorCode return value shall be 0 if operation was succesfull otherwise negated POSIX error code will be re
157. y returning POSIX error code GET_LOCATION_INFO PATH Fetch locationInfo and storagelnfo for given file On success IInfo and sInfo variables will be set On error POSIX error code will be returned interrupting code execution PARENT X Get parent path as string namespace veil namespace client 3 5 veilFuse cc Author Rafal Slota Copyright C 2013 ACK CYFRONET AGH This software is released under the MIT Licence Functions int wrap access const char path int mask 3 4 veilfs cc 39 VeilFS Documentation Release 1 0 int wrap getattr const char path struct stat statbuf int wrap_readlink const char path char link size_t size int wrap_mknod const char path mode_t mode dev_t dev int wrap_mkdir const char path mode_t mode int wrap_unlink const char path int wrap_rmdir const char path int wrap_symlink const char path const char link int wrap_rename const char path const char newpath int wrap_link const char path const char newpath int wrap_chmod const char path mode_t mode 40 Chapter 3 Files VeilFS Documentation Release 1 0 int wrap chown const char path uid t uid gid t gid int wrap truncate const char path off t newSize int wrap utime const char path struct utimbuf ubuf int wrap open const char path struct fuse file info fileInfo int wrap read const char path char buf size_t size off t of

VeilFS Documentation

Contents

Download Pdf Manuals

Related Search

Related Contents