2. Data Analysis and Interpretation Strategies 12 Analysis . Analysis transforms data into findings. . No formula exists for that transformation. DATA DATA EVERYWHERE Guidance, yes. But no recipe. Destination remains unique for each inquirer. . The challenge lies in making sense from massive amounts of data. . Involves: . Reducing the volume of raw information. Sifting trivia from significance. O . Identifying significant trends. . Constructing a framework for communicating the essence of what the data reveals. 13 Raw notes (transcripts, field notes, images etc.) Data Analysis Journey Organising and preparing data for analysis Reading through all data Coding the data Validating the accuracy of information Themes Description Interrelating themes/ description le.g. grounded theory, case study.] Interpreting meaning of themes/ descriptions38 Slide 3/ of 52 In The Back of Your Mind. Ask Yourselves.. How are you framing this study? Without classification, "What is A story I want to tell?", not "What is there is chaos and confusion. THE story?" . Content analysis involves What is happening here? identifying, coding, categorising, What strikes me as interesting? classifying and labelling the How are you telling the story? primary patterns in the data. What should be included, and how much? What is the output? Are you using the correct evidence? 39 Categorizing Example . Similar concepts are grouped together into higher order categories. Business KPis Business KPI Business Innovation Revenue Improvements New revenue Unit Business Customer Satisfaction Product enablement/features . "Big picture" of issues for Profitability New business capabilities irvice/product performance Customer satisfaction Innovation process measures Service/product perform understanding. Business Process KPis Business Process KPI Business Process End-to-end business process Improvements Innovation Scope . Used to reduce the number of concepts the Business Process performance {reliability, Panelency, cost . C provements in business case Contract manufacturing asset productivity) for end-to end business process researcher must work with. nnovation process mean . Example: Business Scope? Operations Performance Project Performance Technical Innovation Availability, reliability On time Platform performance Cost benchmarks and trends On budget improvements I.T./Process/Unit? Innovation? KPI's? On scope Employee trainingt/ certifications Joer satisfaction Innovation process measures formation quality motries Performance Area? Operations? Projects? Operations Projects Innovation . Categorisation helps to differentiate between concepts, Performance Area and enables researchers to identify patterns in the data. 40 Taylor, S. J. (2009]. Acts of conscience: World War II, mental institutions, and religious objectors. Syracuse University Press -Proposed that attendants defined residents according to whether the residents help or hinder attendant's own custodial work Words attendants used to refer to residents Analysis: Attendants' typology of residents "No problem" "Vegetable" "Troublemaker" "Working girl" Control Problems Custodial Problems "Troublemaker" "Low grade" "Cripple" "Biter" "Low grade" 'Soiler" "Puker" "Fighter" "Choker" "Aggressive" "Vegetable" "Working boy" "Wise guy" "Fighter" "Privileged "Pet" "Helper" Character Authority Problems Supervision Problems (PC]' "Wise guy' "Runaway" "Head-banger" "Runaway" "Cripple" "Smart Aleck" "Digger' "Biter" "Puker" "Head-banger" "Choker" "Digger" "Aggressive" No Problems Helpers 'Smart Aleck" "Bucket boy" "Pet" "No problem" "Working girl" "Working boy" "Soiler" "Privileged Character (PC]" "Helper" "Bucket boy"25 Interpretation Interpretation is simply the emergent connection beltween categories and properties based on theoretical codes, and it just happens, because the world is integrated and we are discovering the world - not creating it - Glaser (1992) S. Interpretation Addresses the question: \"So what?\" How do you make sense of what you have learned? What does it mean? > As you interpret your data and create meaning, return to the objective and questions to ask yourselves: 1) What are the relationships between categories, themes and concepts? 7) What patterns have emerged? ) What seems most salient in the data? What is the data telling me? 4) What do | learn by placing the data in the context of existing literature? 5) What do | learn by considering the data through a theoretical lens? ) Using what | have learned, how might | respond to my research question? /s research in the lvory Tower 'Fuzzy, irrelevant, pretentious?\" - Business Week As much as 80% of management research may be irrelevant - Business Week Business academics say nothing in these articles, and they say it in a pretentious way. - Business Week The Nature of Relevant Research > Unless implications are implementable, i.e. prescribed in a manner that could be put to use in practice to resolve a problem, practitioners are unlikely to characterise it as being of interest Ensuring Rigor (Benbasat and Zmud (1999)) Constructs, focus, recommendations etc... should meet expectation/ interest of key stakeholders Must produce cumulative, theory based, and context rich bodies of research Develop frames of reference which are intuitively 3 meaningful to practitioners to organize complex phenomena Portray outputs of research on ways such that it might be utilised by practitioners to justify recommendations Craft research in a clear, simple, and concise manner - accessible to all It is incumbent on researchers in the IS discipline to \"further knowledge that aids in the productive application of IT to human organizations and their management' (ISR 2002) AND to develop and communicate \"knowledge concerning both the management and use of IT for managerial and organizational purposes.\" (Zmud 1997) Treacy (2021) "Mechanisms and Constraints Underpinning Ethically Aligned Artificial 50 Intelligence Systems: An Exploration of Key Performance Areas" European Conference on the Impact of Artificial Intelligence and Robotics Constraints Essential Mechanisms Advanced Mechanisms RQ1: What performance areas Traceability are required for developing Transparency . Competitive advantage . Explainability Promote open standards Configurability Publish algorithms ethically aligned Al? Regulations by governing Shared responsibility Accountability bodies Corporate responsibility . Define responsibilities for User consent for storing each stakeholder . System audits RQ2: What constraints do activity logs organisations experience in Multi-cultural teams Universal system Diversity Contextual awareness Capturing diverse datasets these performance areas to Eliminating bias requirements develop ethically aligned Al? Performance Are Data Integrity . Embrace open.source Data Governance . Privilege exploitation . Data utilisation development . Data owner's consent . Test confidentiality, integrity, & availability requirements RQ3: What mechanisms are used in these performance Access control Security . Evolving threats . Security audits . Identity access management Secure algorithms areas to develop ethically aligned Al? Culture Complacency . Training . Education Proactive user engagement Treacy (2021) "Mechanisms and Constraints Underpinning Ethically Aligned Artificial 51 Intelligence Systems: An Exploration of Key Performance Areas' European Conference on the Impact of Artificial Intelligence and Robotics Traceability: Capturing Diverse Datasets: Universal System Requirements There is always traceability, from " Make sure the dataset you are 'Socioeconomical, geopolitical input to output' (E-1) using has enough diversity and etc. you must train these systems as every culture has differences Explainability: inclusion" (E-4) and you must take those into "You should be able to explain the Privilege Exploitation account." (E-10) system's purpose" (E-6) Shared accounts should be eliminated [E-1) Shared Responsibility: Training "If you are supplying the model, you should Data Integrity Developers should be trained so have accountability and ownership of it. If Before giving the data to the that mistakes should be avoided you are using the model, you should have system, we clean and remove (E-4) the same accountability." (E-3) unwanted data" (E-8) Proactive User Engagement Multi-Cultural Teams: Security Audits "It is in the company's interest to "If you have more diverse teams The key is auditing, we must think about systems proactively, you developing Al, you'll catch biases ensure auditability" (E-4) can't wait to be proactive" (E-2) sooner" (E-5) More quotes, the stronger the argument 52 Writing Your Conclusions . Summarise overall findings . What do we know now, that we didn't . Compare findings to what you before? learned from the literature . How can they use roadmap, model, framework, recommendations you . Discuss your personal view of present? the findings . Limitations often attach to the methods . Discuss how organisation can of study (inadequate sample size, difficulty use the findings? in recruitment etc.) . Represent a weakness that the author acknowledges . State limitations and future Future research research . Remedy some weaknesses in present study Advance new directions for applicationA | So, What Are The Steps to This Journey? Data analysis and interpretation helps us answer the question of \"what does it all mean?!\" Analysis: Summarising and organising data. Interpretation: Finding or making meaning. > Five phases: 1. Data Preparation and Organization 2. Initial Immersion LJ 3. Coding 4. Categorizing and Theming 5. Interpretation i Data Preparation and Organization 1. Data Preparation and Organization First thing to? Prepare the data for analysis! Transcribe interviews, then delete recording. Sensitive material, don't share with 3 parties. Organize for easy access. > Due to the wealth of data to be collected, management, governance, and being able to sort the data is paramount. Create different folders for different interviews. Fill with notes, thoughts, comments or observations based an that interview, colour code important sections. Tips for Data Preparation Important Steps 1. Begin Note and record emergent patterns and Analysis During possible themes while in the field. Fieldwork Add confirming cases to deepen analysis. 2. Inventory Make sure you have all interviews, documents easily on hand. and Organization Check data elements and sources are labelled, dated, and complete. 3. Protect the Back up immediately. Anonymization. Data Delete recording when transcribed. .MASKEDDANCERUK 18 Tips for Data Preparation Important Steps Restate purpose of study, and purpose of 4. Reaffirm analysis. Purpose Be clear about why you are doing this. What is the primary product you are trying to deliver? DUCK 5. Review How have seminal works achieved their findings? Exemplars Use them for guidance. 6. Schedule Analysis requires complete immersion in RABBIT Intense, the data. Dedicated Time It takes time - make time. for Analysis Set a realistic schedule. SEASON 7. Be Open Different people will view things differently. 2.2 Initial ImmersionEliminate the Noise Ever try to locate a clear radio signal through the static noise that fills the airways between signals? Data = noise. Lots of noise. The more data, the louder the noise. The story that detects, makes sense of, interprets, and explains meaningful patterns in the data, is the signal. Your task is distinguishing signal from noise. 2. Initial Immersion Get a sense of the data - allow ideas to form & story to flow Three benefits: Feel the Pulse of the Data Easy to lose sight of the big picture. Gain deep insight into the social worlds you are studying, and what this area of interest can offer. Develop Initial Ideas Everyone should make brief notes on thoughts, ideas and points of interest. Share with group members. Data Reduction Prioritize data for analysis. Avoid paralysis by analysis. What do you have that will answer the research questions effectively? 22 Data, Data, Data.. . The data generated by qualitative methods are voluminous . Sitting down to make sense out of pages of interviews and field notes can be overwhelming . Organising and analysing a mountain of narrative can seem like an impossible task . Example: CDC Study on Community and Scientist Perceptions of Vaccine Trials in the U.S. Extreme Case Obviously 313 interviews 10,000 pages of transcribed text But. on average, 1 hour interview 238 participants could yield 10-15 pages. Range of topics 10 x 1 hour interviews could yield 100-150 pages 23 As You Make Your Way Through Data Collection/Analysis Read, and Re-Read your Know your data inside out Data Field notes, transcripts, documents etc. Keep Track of Hunches, Scribble notes or make a new document. Interpretations and Ideas Insert comments on Word or PDF files for team. Write analytical memos and share with team. Look for Themes that Search through data for emerging themes or patterns: conversation topics, Occur Frequently vocabulary. recurring activities, meanings, feelings etc.. Don't be afraid to identify tentative themes. Move from description, to interpretation and theory. Develop Concepts and Look for similar words and phrases that capture meaning. Theoretical Propositions Compare statements. iii Look for underlying similarities. Develop Models, Useful aids to explore patterns in your data. Frameworks or Theories Sketch out potential relationships for new understanding. 2.3 Coding3. Coding . Concepts that are hidden within textual data. Examine raw data line by line to identify discrete events, incidents, ideas, actions, perceptions and 1. Expected Codes interactions of relevance that are coded as concepts. . Each concept is linked to specific portions of text. 2. Unexpected Codes . Some concepts may be simple, clear and unambiguous. Other concepts can be complex, ambiguous, and viewed differently. . Concepts can be named by researcher or taken from 3. Codes of Unusual literature. . Once identified, they can code the remainder of data. or Conceptual . Keep looking for new concepts while refining old concepts. Interest . Identify characteristics. . Size, colour, level etc.. to group them together. 6 6 Codes are words, or short phrases that capture a summative, salient, essence-capturing, and/or evocative attribute for language-based or visual data. Coding identifies data as belonging to, or representing some type of phenomenon (concept, belief, action, theme, cultural practice, relationship etc.). Examine the data and assign words or phrases that capture their essence. 27 3. Coding . Coding allows you to reduce and classify data generated. . Assign a word, or phrase to segments of data, capturing the essence. . Numerous approaches: 1) In vivo coding O Relies on participants exact language to generate coding 2) Descriptive coding O Mainly uses nouns to summarize segments of data 3) Values coding o Focuses on conflicts, struggles and power issues . Which is the best approach? Justify it!28 Slide 27 of 52 3. Coding Example Interview Transcript In-Vivo Coding Descriptive Coding Values Coding I do not like looking at data o Dislike breaches and feeling O Data breaches Data breaches o Dislikes feeling helpless helpless. Helpless Knowledge important I am a security consultant o Security consultant o Security consultant to job role and would love to feel good about how I work. O Suffers mismatch between knowledge Part of what makes me Suffer mismatch o Malicious actors and ability suffer is the mismatch of What I know what I know, and the ability 0 Ability of malicious actors o Malicious actors' of malicious actors. capability 29 Coding Example - In Vivo Coding Code Abbreviation Description . First Cut? Re - Innovation Reactions to innovation Read through all transcripts Re - Processes Reactions to processes together. Make comments in the Obs - IT Observations of operations impacting IT margins. Obs - Innovating Unit Observations of innovation impacting business unit Post it notes with notions E.g. - IT Example of IT systems about what can be done with E.g. - Pr Example of business processes different parts of the data. Start the coding process. E.g. - Output Effects on processes/output Inn Examples of innovation 30 Coding Example - Descriptive Perceived Importance of Social Ties within "Trend Micro" Solver Solver o, for me, it is not only the money, its just to by that yeah, I had an idea, all the people are On being asked about how important Treacy, S., (2016) "IT-Enabled Innovation Social ties) ended up being huge. saying "its great, why don't and how do you solver interaction is) That is vital. Contest Platforms: An Exploration of the he other guy that I am working on with this (Having a previous relation et is in Munich.. So we were talking in hat in why it is easy for me to gull the Impacts and Mechanisms of Social think it's definitely (important for) that amber I think, so when we finished the it, which is something we I think thi test, and then you say "ok, lets start year are trying to push as well. Even if 1 aborating". don't work in this area, talk to them and Capital" learn from them. [Doctoral Dissertation, University Even if I don't win the contest. I'm in touch We were able to work with some with people that are interested in how it works the Philippines, guys in Taiwan, guys in College Cork]. |South America, guys in Dublin here. er KDM), she didn't care if the idea I didn't have to bring all these people in RQ1: What are the impacts of social a project because for the last six years ] am talking to them on the floor capital on innovation contest platforms? Social Ties wveryday. Trust So that's what I mean, all this social stuff that I have built up for the last six years, I am now reaping the rewards. RQ2: What are the mechanisms used in Reciprocity innovation contest platforms that enable Self-Identity the development of social capital? Shared Language pletely redesigning. were mainly trying to accelerate (social ing more of a culture. Shared Vision myposts of Social Ties within Competitive Markets rog JANALYTIX Crowding NineSimms 31 noCentive solvers are said They can do it alone This is an intervoting place for not only I think z is important that you have Innocentive has maybe in a team and if mo an innovation If the need is most too curry networking, but also making a bunch sammie kind of co-creation boom of money as well. hour petit of slew is that the ce thoma god building the shortlated and they bare to develop creation senates better somuchs. y straggled to allow something, to provide a goosetype, then it people to work in teamin. would be in a seam. managing moore google ets Competiti Example I be a group of $ that are What is commnets of Social Ties within Competitive Markets now JANALYTIX Crowding Innocentice 31 Innocentive solvers are solo They can do it Slide 30 of 52 This is an intersiting place for act only I think it is important that you have and Innocentivs has Quote orking, but alam struggled for years to crea We facilitate connections promotions and building the of money as well. our point of these is that the en creation creates better results. they struggled to allow rowd something, to provid muce time by people to work in teams would be in a team. Code Collaboration Collaboration Collaboration Example Competition/Collaboration There could be a group of s that are a tears on a platform that are abounced is the winner, and then what is common sergis apart from 10,009 euro for the first prize We have more than 100,860 It campgods how much the Workin 2: Quota hallenges, is that they are competing. reward, but the problem is if the It is really an Individual la principle, you great, cooperation is the mpany will start implamar effort. nce, chemistry, physical best, But in praition, a little bit petitions am always a great other ones, so if they cannot advanen and all five people want to be a par once, engineering technology challenging later of ylaiding better results, wefor somebody else who finishes discussions, I think there could be was problems in the long terms What's prospect. Code Competition Competition competition Collaboration Collaboration be start up and spin off king how we can being in The whole design of companies, but allo public these kind of (social thes) activities in the Being ector related to research areh centers future out of course we have to chunk We are not really talking about the On our platform, every project is held would say the same for Idea and also a big crowd of what conditions, what they present individuals, let's say inventors, there is an building of communities deliver the best soheions inthets, whoever in active on exempt those in the groups, but very ncourage social Interaction baring technology and ovation and willing to dual with them, if they work on some kind we have more projects that are Said? of solution together and thay greats song tha solvers. connect with new partners and omething great. how can we divide the do collaborative projects for result between them and our client. innovation, Competition Collaboration Collaboration Competition idual or team, but they work Independently. They have of course the We provide the engineers opportunities for cullaboration That sense of community, we don't need freedom to form teams which is rather different Cola Competition/Collaboration Collaboration Time to sprinkle some codes... 32 Appirie (TopCader) Battle of Chrisper NASA Tournament Lob Minitel Stance Part Phoneamind We alas have what is tugs, at Top Cater for example lee Top Coder Open, and Can the platforum itself. But when I oxenas the that this is, in a mustlong activation of the " Interaction. It was too high means anice tie Fos works mute or less There will be wer so takes the bout coders in outside that they saw Brian you road some sort of prost Quote the world based on the the top contributors wooout to identify you as and they run lie at least profits, they can write atings and the private messages, they performance in different they can form a private in each challange Rate hair fellow that people with state kind of a chat option or er together. Platform. ambers, and puts the facilitator and make shiable thing for them Cule Offlow Events Offine Evalits olver Fruithe Moderators Solver Fromhe How we try to (develop We are wachling with pital lies) was mally the widen is outstanding. and mach other over drinks . Friend', so baied on my I am constaged that Example 3: want to work with that you want to have a And we are also thinki you can feel the tension training where you tsu what I am doing, we can canIt works like a forum.. it will say The Best basin They mas stimul in real life, how they ear other make a match is where they can write with other but as far as 1 brite that is more of a and building and they They can the above the bags and trus if they have an target with that. Same you think that they have meeting, how you whobenge comin up. just won a million dollars breathe breathe and spank if you what sort of logue are on stage, these bind me but people who f things light be interested in my Interviews, Code Offline Events Offline Events Solure Stated Making Selmer Match Making Soher Profile Working with other wonders of the They can form beams, There can be emails, " they can do public reaction with tral Different might want to use, either facilitators or you ke the thening, who manage you are in the system trum forumarea whip up goups or whatever to get the successful etc. the bogdaning, i that is their own awesome that will be a very important next step for us. Research Cade Offline Event sevasion Forum Solver Profile he Forum Solver Match Making Question 33 Example 4: Visualise Your Codes Comments to fall under he cateogories of Mechanisms Useit Impact of Comenut Reciprocity Shared Vision So those are the three categories of people, Shared Language there are people who are working The main challenge, a very clear to we can say there is a certain prediction in your somewhere else but want to do this So, all of these and you know you can so even if you for example, we want them to challenge, the biggest one. is you me because they enjoy solving these problems, allow and understand what they write a blog for us lets cay, they do it already It also depends a lot on the kind of people model, ante they submit a model and the way the communicating. in a normal second is the students who want to test we've done in the past we have on various forms and you will find blogs that you are getting involved In this case, we prediction dota works is you compare their communication environment when Their skills and the third is the nate it very easy for them to spike have a bunch of mathematicians, and they are predictions to the actual values, and see what the you are working on a project or a their profile with Linkedin, So that slightly different breed. it is like you guys differences are, and compare that with thousands complex problem, you get to have across, apart from just the fact that they they don't need to fill in a bunch of write white papers, talk about their ap right, satisfaction through research and just of different instances and come up with a number everat back and forth interactions enjoy the challenges, is that they are details, they just need to click a when they won, give them more visibility as the fact that they get to work on interesting newn the simplest m method is called root mean in a one on one basis to deem the burton and they get all of the petalis well and hopefully get them engaged more stunt square error, which we automatically calculate on problein statement, and then, work contacting and competitions are always s on other aspects and so on. You can't great motivator of yielding better results have that luxury here. Impact of Comtrust henium thed it is a broadcast message and we In fact we want to grow that aspect of Same goes with the learning guys, They are truggled initially. The biggest the platform, where today the here to, so one thing we are enabling is we These people are of stark variety, one reason challenge from our perspective as a primary activity is competitions, but are opening up all of our past competition that large companies is struggling to hire business even has been how do you On our platform, every project is held as a what is going to happen is it we and the data around them, and creating a them is that even though the salaries are So the moment you submit your response, you will get a score for how accurate your model is The take a business challenge which the data competition, where 150+ people able more collaboration between more handheld type of learning competing decent, it is mostly because let's say it the these folks and general discussions, where the exact same objective is there, but end up joining a telecom company, they will moment you get your score, you also go on the client has figure out the best way of compete against each other to deliver ning that into very well defined best solutions. the chances that they will becam we grab hold of one of the top tun who won end up just doing telecom work or perhaps Asible to the world so now data problems". because if you can't platform longer will go up, so that the competition and get them to kind of train just creating one model and then visualizing it becomes a real competition, it is a liam them through step byated that and massaging that for that and massaging that for the rest of their competition. do that, you don't get a lot of want to evolve going on how the problem can be solved and how it liver That is something they don't ws opportunity, you are not talking one forward. on one with each of these 150 guys can be done well. So if your output expectations are not absolutely clear that is an issue Perlevent Importance of ConstructTips for Coding Eight Steps in the Coding Process Eight Steps in the Coding Process 1 Everyone needs a sense of the whole Read all transcnpts carefully, jot down ideas Pick a transcript (shortest, most interesting. top of the pile. whatever) Go throughit Ask yourself \"What s this about?\" Don't think about substance, but underlying meaning Attach comments where needed Complete for several participants and make a list of all bopics Cluster similar topics Forminto columns like: Major, unique, leftovers Go back to data Abbrewviate topics as codes and write codes next to appropriate segments of text See If new categories or codes emerge 2.4 Categorizing and Theming 5 Find the most descriptive wording for your topics and turn theminto categories Reduce total number of categories by grouping topics that relate to each other Draw lines between categories to show interrelationships Make a final decision on the abbreviation for each category and alphabebize these codes Assemble the data belonging to each category and performa preliminary analysis IF necessary, recode your data 4. Categorizing and Theming Once coded, time to look for patterns and relationships. Categorizing = grouping similar or seemingly related codes together. As you study your codes and categories, what themes emerge? Theme = extended phrase or sentence that signals the larger meaning behind a code, or a group of codes. Share: Summaries, Descriptions. Ideas. Key quotes. Interpretive ideas for how codes and categories are related. What you think they mean or add to the story you are telling