Good communication is just as stimulating as black coffee and just as hard to sleep after. - Anne Morrow Lindbergh In May 2021, David Black, CEO of Blackbox, ended his Zoom call with a sense of contentment. Saurabh Sardana, his company's Chief Operating Officer, had just informed him of the successful trial of a newly developed conversational platform using artificial intelligence (AI) and machine learning (ML) to provide on-demand, customised market inteligence. Still in the pilot stage, this new data-driven storytelling technology would soon join Blackbox's suite of data-driven solutions for businesses. Established in 2000 , Blackbox provided decision science solutions to various clients, including government agencies, financial institutions, and technology companies in Singapore and around the region. From opinion surveys to complex market analysis, much of Blackbox's research generated vast quantities of data. While the company's dashboards and reports provided a glimpse into that data, these tools did not necessarily offer insights that would help users make immediate datadriven business decisions. In some cases, the dashboards themselves were not user-friendly and intimidated users even before they could think about how to leverage the information. With these difficulties in mind, Blackbox set out to make data exploration more user-friendly and interactive. This meant making data navigation much more intuitive, allowing even non-specialised users to access data-based insights. It also meant making the navigation a much more conversational experience, introducing the ability to use natural language instead of more 'mechanical' prompts and commands. However, Blackbox was also conscious of the many poorly designed commercial chatbots out there and did not want to add to the long list of dampened user experiences. The objective was to build a more holistic conversationa! data platform to help clients look for relevant data through an easy-tofollow dialogue system attuned to the human mind. Blackbox worked with a local Singapore-based software company, Pand.ai, to develop a chatbot that would enable clients to engage with their data, no matter its quantity, type, or source. Two key challenges emerged: first, the amount of training that the conversational platform needed to understand human language and its hyperlocalised nuances. Second, the need for the system to align with Blackbox's own overall goals, vision, and strategy - ensuring the solution was not just another chatbot, but a valuable business asset. How could Black go about converting the concept to reality? This case was written by Professor Tamas Makany and Lipika Bhattacharya at the Singapore Management University. The case was prepared solely to provide material for class discussion. The authors do not intend to Wlistrate either effective or ineffective handling of a managerial situation. The authors may have disguised certain names and other identifying information to protect confidentiality. Blackbox The founder of Blackbox, David Black, emigrated from Australia to Singapore in 2000. A lawyer by training, he had previous experience in political polling across Australia and New Zealand. Black relocated to Asia as he was convinced that the region was rife with opportunities, with market research and analytics beginning to play a vital role in Asia's growth. Establishing Blackbox in 2000, his focus was on servicing four business areas: business analytics, business strategy, communications/content design, and technology. Based in Singapore, Blackbox aimed to provide clients with decision science solutions that offered consumer, business, and community-wide perspectives on contemporary problems and challenges. It set out to monitor emerging trends regionally and globally, aiming to signal potential changes of significance before they occurred. For over 20 years, the firm had gathered data, constructed advanced analytical approaches, and developed a deep understanding of Asia and its diversity. As the region transformed once again in the 2020s, Blackbox aimed to help its clients access the knowledge they required to make the right decisions. Initially identified as a market research firm, Blackbox had since derived much of its business from its expertise in optimising knowledge management systems. However, a key challenge was that such systems came with a huge amount of data (the data produced by thousands of market surveys conducted every month), and customers did not have time to sieve through it to find what is relevant for their business objectives. Black elaborated, Building knowledge management systems is very important. However, it comes with a lot of information and data. People need deiails immediately at their fingertips and do not have the patience to search for relevant information from a massive knowledge management system. Chatbots and Conversational Al By definition, the technologies that allowed humans to talk to computers by teaching them to decipher and recognise human language were collectively referred to as conversational AI. It encompassed social media chatbots, web and mobile chatbots, virtual assistant devices, virtual agents, and voice recognition agents assisted by AI - in short, anything that responded to human prompts and commands, mimicked human conversations and interactions was referred to as Conversational AI. The chatbot market was valued at US 17.17 billion in 2020 and was projected to reach US $102.29 billion by 2026 . 2 Another study had found that the operational cost savings from using chatbots in banking would reach US $7.3 billion globally by 2023 , up from an estimated US $209 million in 20192 The insurance sector, for example, was expected to benefit from Al, including chatbots, with cost savings of almost US $1.3 billion by 2023 across motor, life, property, and health insurance (up from US $300 million in 2019). 3 These estimates showed that AI-based chatbot platforms were just the tip of a potentially lucrative sector. As chatbot technologies continued to advance, they enabled companies to create and 1 Mordor Intelligence, "Chatbot Market - Growth, Trends, Covid-19 Impact, and Forecasts (2021-2026)", https:i/www.mordorintelligence.com/industry-reportsichatbot-market, accessed Feb 2021. 2 Juniper Research, "Bank Cost Savings via Chatbots to Reach $7.3 Billion by 2023, as Automated Customer Experience Evolve", DATE? https://www.juniperresearch.comipressipress-releases/bank-cost-savings-via-chatbots-reach-7-3bn-2023, accessed Feb 2021. Ibid. 2/9 Permissions hbsp.harvard.edu or 617.783.7860 manage one-to-one customised relationships with existing and potential customers. And they did so with cost efficiencies that were sometimes difficult to imagine or quantify. In 2020, the jury was still out if chatbots were to meet the high expectations set for them. According to Gartner Research, chatbots were at the peak of inflated expectations as of 2020.4 Early publicity around chatbot services showed several success stories as well as innumerable failures. It was predicted that many chatbot providers would leave the market, and abeut 40% of chatbot/virtual assistant applications launched in 2018 would likely be abandoned by 2020.5 With this in mind, Black and his team were clear; they did not want to build yet another chatbot but were looking to develop an application that enabled natural, human-like conversations with their customers. There had been several success stories of conversational Ai implementation. For example, Microsoft launched an AI system called Xiaoice in 2014, which reported a reach of 660 million customers and 450 million third-party smart devices giobally in 2019 . Despite notable initial failures, such as Tay, the offensive social media bot 6, the technology eventually found its way into finance, retail, automotive, real estate, and fashion. In 2017, a popular convenience store chain in Japan, Lawson, worked with Xiaoice to conduct an internal test for potential customer groups. As a result, the chatbot lured more than 40% of its users to the stores to complete a purchase within three days. As the second-largest convenience store chain in Japan, Lawson had more than 12,600 stores in Japan and 22 million users, serving almost 20% of Japan's population. Using manual methods to complete the sarne work would have cost the effort of at least tens of thousands of people. Instead, the cooperation between Lawson and Xiaoice involved discount coupons and daily promotional product advertisements. 7 In 2018 , the chatbot was subsequently spun-off as a separate entity in Japan, Indonesia, India, and the US (named Ruuh in India, Rinna in Japan and Indonesia, and Zo in the US). The entities licensed the technology from Microsoft, while the latter remained a shareholder in the new firms. 8 Another early mover in the chatbot space was the French cosmetic giant Sephora, which launched a chatbot on Kik messenger in 2017. This chatbot had conversations with customers, providing makeup advice, tutoriais, and product suggestions. Seeing the pilot's success, Sephora added several features, inciuding finding the retailer's physical locations and booking appointments for specific services. Again, the results spoke for themselves: customers who had a conversation with the chatbot before booking spent US $50 more in the store compared to non-chatbot users, and overall customer bookings increased by 11% for the year after the launch of the new chatbot service. 9 4 Laurence Goasduff, "Top Trends on the Gartner Hype Cycle for Artificial Intelligence", Gartner, Sep 12, 2019 , https://www.gartner.com/smarterwithgartner/top-trends-on-the-gartner-hype-cycle-for-artificial-intelligence-2019/, aceessed Feb 2021. 5 Ibid. " Kenji Farre, "Remembering Microsoft's Chatbot disaster", UX Planet, Oct 14, 2020, https:/luxplanet,orgiremembering-microsoftschatbot-disaster-3a49d4a633if, accessed Feb 2021. " Camel, "XiaoIce's ninja team, what kind of business model did she start in Japan?" ProgrammerSought, Oct 2017, https://www.leiphone.cominews/201709/swl16Bp3cqIcHLdV.html , accessed Feb 2021. 3 Louis Stone, "Microsoft spins out Chinese AI assistant Xiaoice", AI Business, Jul 13, 2020, https://aibusiness.com/document.asp?doc id=762364 , accessed Feb 2021. "ow Chatbots are Driving Digital Transformation", NEC, Feb 28, 2020 , https://www.nec.comien/global/insights/article/2020022512/index.htm1, accessed Feb 2021. 3/9 This document is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Nov 2023. Copying or posting is an infringement of copyright. Permissions@hbsp.harvard.edu or 617.783.7860 SMU-22-0008 Blackbox Likewise, the US rail service company Amtrak used chatbots to encourage more people to consider trains as an option in a country dominated by air and private ground transportation. To create a better customer experience, the company created a chatbot named 'Julie' to interact with customers. Julie chatted with prospective customers, answered their questions, and suggested the shortest route to a specific destination. It then directed customers to a webpage where they purchased rail tickets. As a result, Amtrak had reported US $1 million savings in customer service costs in the first year, a 25% increase in overall bookings, a 30% increase in online ticket sales, and a 50% increase in communication with customers, year-on-year after the implementation of the chatbot. 10 In Singapore, insurance company Tokio Marine launched a chatbot named TOMI in 2017. In its first project, TOMI was incorporated as an in-built gamification module that allowed agents to win an 'Ang Bao' (a traditional Chinese custom of red packets containing money) each time they correctly answered all the questions asked by the bot. Tokio Marine achieved about an 80% response rate for the campaign. The number of people who interacted with TOMI was consistently at 98% in the first three months. The company later used TOMI as a help desk and agent engagement tool to push relevant articles, news, or tools to the agents. The services of the chatbot were later extended to financial advisors within the network. 11 Information about chatbots like the above helped the team at Blackbox realise that it was necessary to understand the kind of interactions its audience would typically be interested in having with its platform. So how could the solution interface in a way that it surfaced the information the audience needed effectively? The goal was to solve the business needs of Blackbox's clients with insights from huge troves of survey data by enabling conversations in everyday natural language. If a client wanted to understand what the survey results showed about the changes in population sentiments towards the job market in Singapore, they would initially ask an analyst at Blackbox simple questions, such as "Did Singaporeans worry less about losing their jobs in 2019 than today?" The analyst then had to run a data query and call the client back to share the results, such as "In 2019, X\% of Singaporeans agreed with the statement that there would be sufficient job opportunities in Singapore over the next 5 years. Today, there are Y\%." During these calls or data sharing meetings, additional questions drilled further down to more specific industry segments, occupation types, or different periods that sometimes required the analyst to run new queries and repeat the process. While this was tedious for a human analyst, Blackbox's success often depended on how much they anticipated the client's additional questions and provided them with an engaging narrative about the data. They often prepared multiple data scenarios and hoped that the client would ask questions for the prepared answers. Such turn taking was more naturally suited for a conversational AI programmed to instantaneously extract insights from big datasets and translate them into a natural language format. The company hoped that an intelligent conversational data platform would reduce the frequency of client-analyst phone calls and improve the client's experience. As Black noted, [W]e started contemplating developing a chatbot to create a better suite of tools for our clients. But the concept of a chatbot is that you talk to some sort of agent. We did not want an agent; we 10 Ibid. 11 "Do chatbots really work for insurance?" Asia Insurance Review, Oct 2019, https://ww.asiainsurancereview.com/Magazine/ReadMagazineArticle?aid-42633, accessed Feb 2021. 4/9 Permissions@hbsp.harvard.edu or 617.783.7860 were rather looking for a solution to help engage our customers. So we started to wiggle with ideas at that point. Initial discussions on conversational design By 2018, Blackbox was expanding rapidly, as client demand for data-driven insights increased and the evolution of 'Big Data' drove organisations to rely on data to influence business decisions. The company realised that it required a Business Intelligence (BI) system, so it implemented Salesforce to create meaningful content from the growing data that it produced every day. 12 Blackbox also invested in interactive data visualisation software, Tableau, to help present data more impactfully. The power of BI allowed the firm to create custom dashboards for its clients. However, the company quickly realised that dashboards had become common in the market and ceased to be a differentiating factor for potential clients. Saurabh Sardana, the Chief Operating Officer at Blackbox, noted, We were less focused on tools and more on intent. The intent was that we are in the data business, but data (by itself) doesn't excite many - numbers are boring. Our original goal was to bring back that excitement in data, inject life into data and add more context and meaning into it. Sardana and Black believed that human-centred conversational AI that turned data into insights might be the exciting novel thing needed, creating tangible business benefits. However, for a small firm, a large-scale technical project rcpresented a huge risk. Resources were limited, and it was difficult to justify diverting them towards experimental innovation. Given that their clients were mostly government agencies, there was also a concern about data confidentiality. As they became aware of their constraints, they decided to start with a pilot project and scale up gradually if the solution worked. More importantly, this evolutionary approach allowed them to incorporate a variety of client-specific needs as they emerged. After several brainistorming sessions, the management team was able to fine-tune its vision. The goal was to distribute data insights to its clients in a non-technical, customer-friendly format. The team initially targeted professionals who made policy decisions based on the data analysis responses that the platform provided. They wanted the user experience (UX) to engage these clients in a conversation modeled after the client-analyst phone calls. The team decided to apply the Natural Conversation Framework (NCF) for their design process (refer to Exhibit 1). They set out to build an environment-agnostic, on-demand, AI-powered, conversational data platform that clients could use to receive data insights refined through multiturn, natural language conversations. Ideation of the solution In 2019, Black and his team realised that a significant challenge in making data engaging for their clients was a lack of innovation in delivery formats. PowerPoint, the most used tool for research 12 Business Intelligence (BI) comprised the strategies and technologies used by enterprises for the data analysis of business information, referenced from N. Dedic, C. Stanier, "Measuring the Success of Changes to Existing Business Intelligence Solutions to Improve Business Intelligence Reporting", Springer International Publishing, 2016, pp. 225-236, accessed March 2020. 5/9 This document is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Now 2023. Copying or posting is an infringement of copyright. Permissions@hbsp.harvard.edu or 617.783.7860 SMU-22-0008 Blackbox presentations, was good to condense or frame a story but insufficient to tell a complete, engaging story. Sardana noted, The industry was facing a state of PowerPoint fatigue. Even dashboards provided an overwhelming amount of data, which was becoming less engaging for people. Dashboard adoption levels were low and did not help in democratising data. More brainstorming sessions followed to figure out specific details about the platform. They involved raising questions like: "What are the primary use cases? What client profiles should be targeted first? What are the likely scenarios that a client would turn to a chatbot instead of a human analyst? How could this new technology be positioned to present a competitive advantage?" While the original idea was to build a more holistic platform, the discussions helped the team realise that they would need to start small and see the results from pilot tests. There was a need to offer clients a simple way to interact with the data even when analysts from Blackbox were not in the room. The company also conducted extensive open-ended ethnographic interviews with their clients to understand their pain points and motivations with their data strategy. Black recalled, When we spoke to clients, we were much more speculative. We asked them questions like "what do you need" and "what if we could add these features to this experiment to add value." To me, the idea was not hugely enticing at first, but it sounded better than a simple chatbot. I kept thinking that it would cost me a significant amount to build. We were trying to understand people and translate that into a solution, which was not going to be easy. In March 2020, Sardana kicked off the ideation with a simple sketch presentation (refer to Exhibit 2). He described this as a "visualisation of what the chatbot would look like, and that allowed us to align our different objectives." As the company did not have an internal AI expert, Black hired a consultant advisor to fine-tune the requirements and look for a venclor to build the solution. First, the consultant shortlisted eleven different companies that could suit Blackbox's needs. Then, based on expertise, timelines, and costs, they narrowed the search to two companies with viable product and technical capabilities. One of the two finalists was a large technology company in the Ukraine with a broad talent pool and expertise in building AI solutions across Fintech, multimedia, retail, travel, and the telecom industries. The other vendor - Pand.ai 13 - was a Singapore-based AI start-up specialising in deep learning and Natural Language Processing (NLP) with experience in working with Fortune 500 companies in the finanicial and insurance segments. Pand.ai proposed a three-week pilot project for a custom-tailored chatbot. Sardana summed up the vendor selection process, While there were several other competent vendors, we found Pand.ai to offer the most value with customisation for our needs. The other vendor had sketched out a much bigger scope project, so we concentrated on finding the vendor who best fit our needs. By mid-June 2020, the Blackbox and Pand.ai teams started negotiating the terms of a pilot contract about a chatbot prototype and estimated costs of a subsequent full implementation that included talent, organisational and technical requirements. Early on, however, the teams realised that they had an understanding gap. Specifically, Blackbox lacked the technical expertise to develop conversational AI, and Pand.ai had never worked with a client who used large-scale surveys as a 13 Pand.ai, www.pand.ai, accessed April 2022. 6/9 This document is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Nov 2023. Copying or posting is an infringement of copyright. Permissions @ hbsp.harvard.edu or 617.783.7860 SMU-22-0008 Blackbox primary data source. As a result, they had to rely on the consultant to translate non-technica! business needs into technical requirements. After a week of intense negotiations, the parties signed a contract that defined the new Blackbox conversational data platform that married market research survey insights with the simplicity of everyday natural language. Sardana recalled, We wanted the user interface to be conversational, like WhatsApp, Slack, and other chat applications, so that it became a conversational platform which was an enhancement over the current dashboards that we were using. Prototyping The main deliverable of the pilot phase was a prototype chatbot with limited capabilities focused on a single client with specific use-cases. The final product requirement document specified the following key assumptions about the prototype: - Deployment in one of the available channels used by the client (either Slack or Web) - User input limited to text-based format only (no voice input yet) - Hyperlocalised language model and NLP engine to understand human user intent - Interface with the Blackbox dialog content (training) database - Statistical calculation of data insights based on algorithmic models - Output the data insights in a multi-turn, waterfall natural language dialog format - Basic data visualisations with limited card types (charts \& graphs) - Basic capabilities to measure client engagement using telemetry. The prototype development followed an iterative agile process. The process took three weeks, with weekly email updates, regular phone calls, and Zoom meetings with product demos during the final week. Training the algorithm of the conversational AI was one of the most challenging tasks. The team selected a compact subset of 18,000 entries from a large survey that contained all the test scenarios and data structures. They also prepared a content strategy document modeled on real-life conversations between Biackbox's data analysts and their clients. This document helped shape the prototype's conversational style, personality, and tone of voice (refer to Exhibit 3 for a Sample of training dataset). In order for the prototype to understand the human users' input, it needed to utilise Pand.ai's proprietary language model and an NLP engine. As the selected client for the pilot was based in Singapore, the language model was hyperlocalised to handle local colloquialisms, while the NLP processed general syntax (grammar), semantics (meaning), and pragmatics (entities and intents) of the users. The engine included two main components: (1) Intent Classification to match user input/questions with the intents in the database at a specified confidence score, and (2) Named Entity Recognition to perform information extraction by capturing entities such as date, time, survey questions, product types, etc. For ambiguous intents, the prototype asked further clarifying questions. The conversational interface supported various response formats, including text, images, and buttons. In the simplest 7/9 is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Nov 2023. Copying or posting is an infringement of copyri terms, it answered people's questions about survey data using everyday natural language and simple data visualisations (refer to Exhibit 4 for visuals from the final prototype). Evaluation of the prototype The prototype was evaluated first internally by the Blackbox analysts and then by selected client users. The critical measure of success was tied back to increased platform usage with the new chatbot over existing dashboards. This evaluation led the company to ask itself a few additional questions: Did it make business sense to build the chatbot? Did the benefit of the solution justify the costs to develop and maintain the platform? How would the client and internal stakeholders perceive the solution? Was it innovative? Was it useful? The initial prototype went through several iterations, including two rounds of internal testing and two user-acceptance tests. After that, there was an additional independent test by the client. One of the critical technical learnings was the need for a scalable process to update the ever-changing database structure that the chatbot interfaced with. For the final rollout, this process needed an automated application programming interface (API) that could handle dynamic changes in the structure of such large databases. After the successful pilot, the team at Blackbox started discussing a full-phased implementation. The subsequent phases of deployment fosused on scaling up the prototype. The intention was to keep a human-centred approach based on conversational UX design principles that could continue to evolve based on feedback from business clients. What Next? Implementing AI-based technology involved significant investments and took time to deploy and achieve the desired efficiency. But like many small and medium enterprises in Singapore, digitalisation of products and services was no longer an option; it was a must-have. This was especially true as the assimilation of AI capabilities was quickly rising in the market research industry, and companies were starting to automate many of their capabilities, including introducing automated voice-based surveys. 14 Blackbox was under immense pressure to evolve its offerings in this changing environment and do so ahead of competitors. As a result, they had to either innovate rapidly to stay competitive or risk stagnation. More than anything, the process had proved to be pivotal for the company in several ways. First, it had given them more confidence to pursue customised technological solutions with the help of skilled partners. Second, it had expanded the company's sense of product development to serve their clients better. Finally, the process was transformational for Blackbox insofar as it engaged key personnel in the company and gave them a glimpse of the future. Rather than talk about change, the experience was insightful and resulted in several staff members thinking about the business in a new way. The result was not just a new chapter for Blackbox but also an opportunity for a whole new story. 14 How AI will reinvent the market research industry, Qualtrics Experience Management Report, Qualtrics, Aug 2018, https://www.qualtrics.com/m/assets/wp-content/uploads/2018/08/AI-in-MR-Final.pdf , accessed Feb 2021. 8/9 This document is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Nov 2023. Copying or posting is an infringement of copyright. Permissions@hbsp.harvard.edu or 617.783.7860 EXHIBIT 1: NATURAL CONVERSATION FRAMEWORK (NCF) Natural Conversation Framework (NCF), developed at IBM Research, is a systematic framework for designing interfaces that work like natural conversation. "Conversational" means a natural-language interface that both recognises common conversational actions and preserves the sequential context of previous turns across future turns, so the agent can respond appropriately. The NCF provides a pattern language of generic, reusable conversational user experience (UX) patterns that are independent of any particular technology platform. The patterns are simplified forms of natural human conversation patterns documented in the Conversation Analysis (CA) literature, for example, those of sequence organisation or repair. The NCF consists of four main components: 1) an interaction model of "expandable sequences," 2) a corresponding content format, 3) a pattern language with 100 generic UX patterns and 4) a navigation method of six basic user actions. Source: Robert J. Moore, Raphael Arar, Conversational UX Design: A Practitioner's Guide to the Natural Conversation Framework, Association for Computing Machinery New York, April 2019, pp. 64 - 65, https://doi.org/10.1145/3304087, accessed Feb 2021. EXHIBIT 2: INITIAL IDEA OF THE CONVERSATION 9/9 This document is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Nov 2023. Permissions@hbsp.harvard.edu or 617.783.7860 What the complexity in blackbox work process? and how the development chatbot changed it? Discuss all of them A) Data B) Design C) Strategy 2) From the strategic point of view, how would this development bb develop differentiatio among other competitors? market segmentation. 3) Given multiple success stories in the case? what lessons can be drawn to ensure the success of BB chatbot? 4) what are the barriers faced by the project team in the design process of Al ChatBot Choose one of them. Technical Strategical Ethical Customers must like the design. 1. Explain the complexity of the black box work process? and how the development of the chatbot changed it. 2. From the strategic point of view, how will this development help BB develop differentiation amongst its competitors? 3. Given multiple success stories in the case? what lessons can be drawn to ensure the success of the BB chatbot? 4. what are the barriers faced by the project team in the design process of AI ChatBot? Choose one of them. 5. What were the research and design methodologies deployed in the prototyping stage? Good communication is just as stimulating as black coffee and just as hard to sleep after. - Anne Morrow Lindbergh In May 2021, David Black, CEO of Blackbox, ended his Zoom call with a sense of contentment. Saurabh Sardana, his company's Chief Operating Officer, had just informed him of the successful trial of a newly developed conversational platform using artificial intelligence (AI) and machine learning (ML) to provide on-demand, customised market inteligence. Still in the pilot stage, this new data-driven storytelling technology would soon join Blackbox's suite of data-driven solutions for businesses. Established in 2000 , Blackbox provided decision science solutions to various clients, including government agencies, financial institutions, and technology companies in Singapore and around the region. From opinion surveys to complex market analysis, much of Blackbox's research generated vast quantities of data. While the company's dashboards and reports provided a glimpse into that data, these tools did not necessarily offer insights that would help users make immediate datadriven business decisions. In some cases, the dashboards themselves were not user-friendly and intimidated users even before they could think about how to leverage the information. With these difficulties in mind, Blackbox set out to make data exploration more user-friendly and interactive. This meant making data navigation much more intuitive, allowing even non-specialised users to access data-based insights. It also meant making the navigation a much more conversational experience, introducing the ability to use natural language instead of more 'mechanical' prompts and commands. However, Blackbox was also conscious of the many poorly designed commercial chatbots out there and did not want to add to the long list of dampened user experiences. The objective was to build a more holistic conversationa! data platform to help clients look for relevant data through an easy-tofollow dialogue system attuned to the human mind. Blackbox worked with a local Singapore-based software company, Pand.ai, to develop a chatbot that would enable clients to engage with their data, no matter its quantity, type, or source. Two key challenges emerged: first, the amount of training that the conversational platform needed to understand human language and its hyperlocalised nuances. Second, the need for the system to align with Blackbox's own overall goals, vision, and strategy - ensuring the solution was not just another chatbot, but a valuable business asset. How could Black go about converting the concept to reality? This case was written by Professor Tamas Makany and Lipika Bhattacharya at the Singapore Management University. The case was prepared solely to provide material for class discussion. The authors do not intend to Wlistrate either effective or ineffective handling of a managerial situation. The authors may have disguised certain names and other identifying information to protect confidentiality. Blackbox The founder of Blackbox, David Black, emigrated from Australia to Singapore in 2000. A lawyer by training, he had previous experience in political polling across Australia and New Zealand. Black relocated to Asia as he was convinced that the region was rife with opportunities, with market research and analytics beginning to play a vital role in Asia's growth. Establishing Blackbox in 2000, his focus was on servicing four business areas: business analytics, business strategy, communications/content design, and technology. Based in Singapore, Blackbox aimed to provide clients with decision science solutions that offered consumer, business, and community-wide perspectives on contemporary problems and challenges. It set out to monitor emerging trends regionally and globally, aiming to signal potential changes of significance before they occurred. For over 20 years, the firm had gathered data, constructed advanced analytical approaches, and developed a deep understanding of Asia and its diversity. As the region transformed once again in the 2020s, Blackbox aimed to help its clients access the knowledge they required to make the right decisions. Initially identified as a market research firm, Blackbox had since derived much of its business from its expertise in optimising knowledge management systems. However, a key challenge was that such systems came with a huge amount of data (the data produced by thousands of market surveys conducted every month), and customers did not have time to sieve through it to find what is relevant for their business objectives. Black elaborated, Building knowledge management systems is very important. However, it comes with a lot of information and data. People need deiails immediately at their fingertips and do not have the patience to search for relevant information from a massive knowledge management system. Chatbots and Conversational Al By definition, the technologies that allowed humans to talk to computers by teaching them to decipher and recognise human language were collectively referred to as conversational AI. It encompassed social media chatbots, web and mobile chatbots, virtual assistant devices, virtual agents, and voice recognition agents assisted by AI - in short, anything that responded to human prompts and commands, mimicked human conversations and interactions was referred to as Conversational AI. The chatbot market was valued at US 17.17 billion in 2020 and was projected to reach US $102.29 billion by 2026 . 2 Another study had found that the operational cost savings from using chatbots in banking would reach US $7.3 billion globally by 2023 , up from an estimated US $209 million in 20192 The insurance sector, for example, was expected to benefit from Al, including chatbots, with cost savings of almost US $1.3 billion by 2023 across motor, life, property, and health insurance (up from US $300 million in 2019). 3 These estimates showed that AI-based chatbot platforms were just the tip of a potentially lucrative sector. As chatbot technologies continued to advance, they enabled companies to create and 1 Mordor Intelligence, "Chatbot Market - Growth, Trends, Covid-19 Impact, and Forecasts (2021-2026)", https:i/www.mordorintelligence.com/industry-reportsichatbot-market, accessed Feb 2021. 2 Juniper Research, "Bank Cost Savings via Chatbots to Reach $7.3 Billion by 2023, as Automated Customer Experience Evolve", DATE? https://www.juniperresearch.comipressipress-releases/bank-cost-savings-via-chatbots-reach-7-3bn-2023, accessed Feb 2021. Ibid. 2/9 Permissions hbsp.harvard.edu or 617.783.7860 manage one-to-one customised relationships with existing and potential customers. And they did so with cost efficiencies that were sometimes difficult to imagine or quantify. In 2020, the jury was still out if chatbots were to meet the high expectations set for them. According to Gartner Research, chatbots were at the peak of inflated expectations as of 2020.4 Early publicity around chatbot services showed several success stories as well as innumerable failures. It was predicted that many chatbot providers would leave the market, and abeut 40% of chatbot/virtual assistant applications launched in 2018 would likely be abandoned by 2020.5 With this in mind, Black and his team were clear; they did not want to build yet another chatbot but were looking to develop an application that enabled natural, human-like conversations with their customers. There had been several success stories of conversational Ai implementation. For example, Microsoft launched an AI system called Xiaoice in 2014, which reported a reach of 660 million customers and 450 million third-party smart devices giobally in 2019 . Despite notable initial failures, such as Tay, the offensive social media bot 6, the technology eventually found its way into finance, retail, automotive, real estate, and fashion. In 2017, a popular convenience store chain in Japan, Lawson, worked with Xiaoice to conduct an internal test for potential customer groups. As a result, the chatbot lured more than 40% of its users to the stores to complete a purchase within three days. As the second-largest convenience store chain in Japan, Lawson had more than 12,600 stores in Japan and 22 million users, serving almost 20% of Japan's population. Using manual methods to complete the sarne work would have cost the effort of at least tens of thousands of people. Instead, the cooperation between Lawson and Xiaoice involved discount coupons and daily promotional product advertisements. 7 In 2018 , the chatbot was subsequently spun-off as a separate entity in Japan, Indonesia, India, and the US (named Ruuh in India, Rinna in Japan and Indonesia, and Zo in the US). The entities licensed the technology from Microsoft, while the latter remained a shareholder in the new firms. 8 Another early mover in the chatbot space was the French cosmetic giant Sephora, which launched a chatbot on Kik messenger in 2017. This chatbot had conversations with customers, providing makeup advice, tutoriais, and product suggestions. Seeing the pilot's success, Sephora added several features, inciuding finding the retailer's physical locations and booking appointments for specific services. Again, the results spoke for themselves: customers who had a conversation with the chatbot before booking spent US $50 more in the store compared to non-chatbot users, and overall customer bookings increased by 11% for the year after the launch of the new chatbot service. 9 4 Laurence Goasduff, "Top Trends on the Gartner Hype Cycle for Artificial Intelligence", Gartner, Sep 12, 2019 , https://www.gartner.com/smarterwithgartner/top-trends-on-the-gartner-hype-cycle-for-artificial-intelligence-2019/, aceessed Feb 2021. 5 Ibid. " Kenji Farre, "Remembering Microsoft's Chatbot disaster", UX Planet, Oct 14, 2020, https:/luxplanet,orgiremembering-microsoftschatbot-disaster-3a49d4a633if, accessed Feb 2021. " Camel, "XiaoIce's ninja team, what kind of business model did she start in Japan?" ProgrammerSought, Oct 2017, https://www.leiphone.cominews/201709/swl16Bp3cqIcHLdV.html , accessed Feb 2021. 3 Louis Stone, "Microsoft spins out Chinese AI assistant Xiaoice", AI Business, Jul 13, 2020, https://aibusiness.com/document.asp?doc id=762364 , accessed Feb 2021. "ow Chatbots are Driving Digital Transformation", NEC, Feb 28, 2020 , https://www.nec.comien/global/insights/article/2020022512/index.htm1, accessed Feb 2021. 3/9 This document is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Nov 2023. Copying or posting is an infringement of copyright. Permissions@hbsp.harvard.edu or 617.783.7860 SMU-22-0008 Blackbox Likewise, the US rail service company Amtrak used chatbots to encourage more people to consider trains as an option in a country dominated by air and private ground transportation. To create a better customer experience, the company created a chatbot named 'Julie' to interact with customers. Julie chatted with prospective customers, answered their questions, and suggested the shortest route to a specific destination. It then directed customers to a webpage where they purchased rail tickets. As a result, Amtrak had reported US $1 million savings in customer service costs in the first year, a 25% increase in overall bookings, a 30% increase in online ticket sales, and a 50% increase in communication with customers, year-on-year after the implementation of the chatbot. 10 In Singapore, insurance company Tokio Marine launched a chatbot named TOMI in 2017. In its first project, TOMI was incorporated as an in-built gamification module that allowed agents to win an 'Ang Bao' (a traditional Chinese custom of red packets containing money) each time they correctly answered all the questions asked by the bot. Tokio Marine achieved about an 80% response rate for the campaign. The number of people who interacted with TOMI was consistently at 98% in the first three months. The company later used TOMI as a help desk and agent engagement tool to push relevant articles, news, or tools to the agents. The services of the chatbot were later extended to financial advisors within the network. 11 Information about chatbots like the above helped the team at Blackbox realise that it was necessary to understand the kind of interactions its audience would typically be interested in having with its platform. So how could the solution interface in a way that it surfaced the information the audience needed effectively? The goal was to solve the business needs of Blackbox's clients with insights from huge troves of survey data by enabling conversations in everyday natural language. If a client wanted to understand what the survey results showed about the changes in population sentiments towards the job market in Singapore, they would initially ask an analyst at Blackbox simple questions, such as "Did Singaporeans worry less about losing their jobs in 2019 than today?" The analyst then had to run a data query and call the client back to share the results, such as "In 2019, X\% of Singaporeans agreed with the statement that there would be sufficient job opportunities in Singapore over the next 5 years. Today, there are Y\%." During these calls or data sharing meetings, additional questions drilled further down to more specific industry segments, occupation types, or different periods that sometimes required the analyst to run new queries and repeat the process. While this was tedious for a human analyst, Blackbox's success often depended on how much they anticipated the client's additional questions and provided them with an engaging narrative about the data. They often prepared multiple data scenarios and hoped that the client would ask questions for the prepared answers. Such turn taking was more naturally suited for a conversational AI programmed to instantaneously extract insights from big datasets and translate them into a natural language format. The company hoped that an intelligent conversational data platform would reduce the frequency of client-analyst phone calls and improve the client's experience. As Black noted, [W]e started contemplating developing a chatbot to create a better suite of tools for our clients. But the concept of a chatbot is that you talk to some sort of agent. We did not want an agent; we 10 Ibid. 11 "Do chatbots really work for insurance?" Asia Insurance Review, Oct 2019, https://ww.asiainsurancereview.com/Magazine/ReadMagazineArticle?aid-42633, accessed Feb 2021. 4/9 Permissions@hbsp.harvard.edu or 617.783.7860 were rather looking for a solution to help engage our customers. So we started to wiggle with ideas at that point. Initial discussions on conversational design By 2018, Blackbox was expanding rapidly, as client demand for data-driven insights increased and the evolution of 'Big Data' drove organisations to rely on data to influence business decisions. The company realised that it required a Business Intelligence (BI) system, so it implemented Salesforce to create meaningful content from the growing data that it produced every day. 12 Blackbox also invested in interactive data visualisation software, Tableau, to help present data more impactfully. The power of BI allowed the firm to create custom dashboards for its clients. However, the company quickly realised that dashboards had become common in the market and ceased to be a differentiating factor for potential clients. Saurabh Sardana, the Chief Operating Officer at Blackbox, noted, We were less focused on tools and more on intent. The intent was that we are in the data business, but data (by itself) doesn't excite many - numbers are boring. Our original goal was to bring back that excitement in data, inject life into data and add more context and meaning into it. Sardana and Black believed that human-centred conversational AI that turned data into insights might be the exciting novel thing needed, creating tangible business benefits. However, for a small firm, a large-scale technical project rcpresented a huge risk. Resources were limited, and it was difficult to justify diverting them towards experimental innovation. Given that their clients were mostly government agencies, there was also a concern about data confidentiality. As they became aware of their constraints, they decided to start with a pilot project and scale up gradually if the solution worked. More importantly, this evolutionary approach allowed them to incorporate a variety of client-specific needs as they emerged. After several brainistorming sessions, the management team was able to fine-tune its vision. The goal was to distribute data insights to its clients in a non-technical, customer-friendly format. The team initially targeted professionals who made policy decisions based on the data analysis responses that the platform provided. They wanted the user experience (UX) to engage these clients in a conversation modeled after the client-analyst phone calls. The team decided to apply the Natural Conversation Framework (NCF) for their design process (refer to Exhibit 1). They set out to build an environment-agnostic, on-demand, AI-powered, conversational data platform that clients could use to receive data insights refined through multiturn, natural language conversations. Ideation of the solution In 2019, Black and his team realised that a significant challenge in making data engaging for their clients was a lack of innovation in delivery formats. PowerPoint, the most used tool for research 12 Business Intelligence (BI) comprised the strategies and technologies used by enterprises for the data analysis of business information, referenced from N. Dedic, C. Stanier, "Measuring the Success of Changes to Existing Business Intelligence Solutions to Improve Business Intelligence Reporting", Springer International Publishing, 2016, pp. 225-236, accessed March 2020. 5/9 This document is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Now 2023. Copying or posting is an infringement of copyright. Permissions@hbsp.harvard.edu or 617.783.7860 SMU-22-0008 Blackbox presentations, was good to condense or frame a story but insufficient to tell a complete, engaging story. Sardana noted, The industry was facing a state of PowerPoint fatigue. Even dashboards provided an overwhelming amount of data, which was becoming less engaging for people. Dashboard adoption levels were low and did not help in democratising data. More brainstorming sessions followed to figure out specific details about the platform. They involved raising questions like: "What are the primary use cases? What client profiles should be targeted first? What are the likely scenarios that a client would turn to a chatbot instead of a human analyst? How could this new technology be positioned to present a competitive advantage?" While the original idea was to build a more holistic platform, the discussions helped the team realise that they would need to start small and see the results from pilot tests. There was a need to offer clients a simple way to interact with the data even when analysts from Blackbox were not in the room. The company also conducted extensive open-ended ethnographic interviews with their clients to understand their pain points and motivations with their data strategy. Black recalled, When we spoke to clients, we were much more speculative. We asked them questions like "what do you need" and "what if we could add these features to this experiment to add value." To me, the idea was not hugely enticing at first, but it sounded better than a simple chatbot. I kept thinking that it would cost me a significant amount to build. We were trying to understand people and translate that into a solution, which was not going to be easy. In March 2020, Sardana kicked off the ideation with a simple sketch presentation (refer to Exhibit 2). He described this as a "visualisation of what the chatbot would look like, and that allowed us to align our different objectives." As the company did not have an internal AI expert, Black hired a consultant advisor to fine-tune the requirements and look for a venclor to build the solution. First, the consultant shortlisted eleven different companies that could suit Blackbox's needs. Then, based on expertise, timelines, and costs, they narrowed the search to two companies with viable product and technical capabilities. One of the two finalists was a large technology company in the Ukraine with a broad talent pool and expertise in building AI solutions across Fintech, multimedia, retail, travel, and the telecom industries. The other vendor - Pand.ai 13 - was a Singapore-based AI start-up specialising in deep learning and Natural Language Processing (NLP) with experience in working with Fortune 500 companies in the finanicial and insurance segments. Pand.ai proposed a three-week pilot project for a custom-tailored chatbot. Sardana summed up the vendor selection process, While there were several other competent vendors, we found Pand.ai to offer the most value with customisation for our needs. The other vendor had sketched out a much bigger scope project, so we concentrated on finding the vendor who best fit our needs. By mid-June 2020, the Blackbox and Pand.ai teams started negotiating the terms of a pilot contract about a chatbot prototype and estimated costs of a subsequent full implementation that included talent, organisational and technical requirements. Early on, however, the teams realised that they had an understanding gap. Specifically, Blackbox lacked the technical expertise to develop conversational AI, and Pand.ai had never worked with a client who used large-scale surveys as a 13 Pand.ai, www.pand.ai, accessed April 2022. 6/9 This document is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Nov 2023. Copying or posting is an infringement of copyright. Permissions @ hbsp.harvard.edu or 617.783.7860 SMU-22-0008 Blackbox primary data source. As a result, they had to rely on the consultant to translate non-technica! business needs into technical requirements. After a week of intense negotiations, the parties signed a contract that defined the new Blackbox conversational data platform that married market research survey insights with the simplicity of everyday natural language. Sardana recalled, We wanted the user interface to be conversational, like WhatsApp, Slack, and other chat applications, so that it became a conversational platform which was an enhancement over the current dashboards that we were using. Prototyping The main deliverable of the pilot phase was a prototype chatbot with limited capabilities focused on a single client with specific use-cases. The final product requirement document specified the following key assumptions about the prototype: - Deployment in one of the available channels used by the client (either Slack or Web) - User input limited to text-based format only (no voice input yet) - Hyperlocalised language model and NLP engine to understand human user intent - Interface with the Blackbox dialog content (training) database - Statistical calculation of data insights based on algorithmic models - Output the data insights in a multi-turn, waterfall natural language dialog format - Basic data visualisations with limited card types (charts \& graphs) - Basic capabilities to measure client engagement using telemetry. The prototype development followed an iterative agile process. The process took three weeks, with weekly email updates, regular phone calls, and Zoom meetings with product demos during the final week. Training the algorithm of the conversational AI was one of the most challenging tasks. The team selected a compact subset of 18,000 entries from a large survey that contained all the test scenarios and data structures. They also prepared a content strategy document modeled on real-life conversations between Biackbox's data analysts and their clients. This document helped shape the prototype's conversational style, personality, and tone of voice (refer to Exhibit 3 for a Sample of training dataset). In order for the prototype to understand the human users' input, it needed to utilise Pand.ai's proprietary language model and an NLP engine. As the selected client for the pilot was based in Singapore, the language model was hyperlocalised to handle local colloquialisms, while the NLP processed general syntax (grammar), semantics (meaning), and pragmatics (entities and intents) of the users. The engine included two main components: (1) Intent Classification to match user input/questions with the intents in the database at a specified confidence score, and (2) Named Entity Recognition to perform information extraction by capturing entities such as date, time, survey questions, product types, etc. For ambiguous intents, the prototype asked further clarifying questions. The conversational interface supported various response formats, including text, images, and buttons. In the simplest 7/9 is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Nov 2023. Copying or posting is an infringement of copyri terms, it answered people's questions about survey data using everyday natural language and simple data visualisations (refer to Exhibit 4 for visuals from the final prototype). Evaluation of the prototype The prototype was evaluated first internally by the Blackbox analysts and then by selected client users. The critical measure of success was tied back to increased platform usage with the new chatbot over existing dashboards. This evaluation led the company to ask itself a few additional questions: Did it make business sense to build the chatbot? Did the benefit of the solution justify the costs to develop and maintain the platform? How would the client and internal stakeholders perceive the solution? Was it innovative? Was it useful? The initial prototype went through several iterations, including two rounds of internal testing and two user-acceptance tests. After that, there was an additional independent test by the client. One of the critical technical learnings was the need for a scalable process to update the ever-changing database structure that the chatbot interfaced with. For the final rollout, this process needed an automated application programming interface (API) that could handle dynamic changes in the structure of such large databases. After the successful pilot, the team at Blackbox started discussing a full-phased implementation. The subsequent phases of deployment fosused on scaling up the prototype. The intention was to keep a human-centred approach based on conversational UX design principles that could continue to evolve based on feedback from business clients. What Next? Implementing AI-based technology involved significant investments and took time to deploy and achieve the desired efficiency. But like many small and medium enterprises in Singapore, digitalisation of products and services was no longer an option; it was a must-have. This was especially true as the assimilation of AI capabilities was quickly rising in the market research industry, and companies were starting to automate many of their capabilities, including introducing automated voice-based surveys. 14 Blackbox was under immense pressure to evolve its offerings in this changing environment and do so ahead of competitors. As a result, they had to either innovate rapidly to stay competitive or risk stagnation. More than anything, the process had proved to be pivotal for the company in several ways. First, it had given them more confidence to pursue customised technological solutions with the help of skilled partners. Second, it had expanded the company's sense of product development to serve their clients better. Finally, the process was transformational for Blackbox insofar as it engaged key personnel in the company and gave them a glimpse of the future. Rather than talk about change, the experience was insightful and resulted in several staff members thinking about the business in a new way. The result was not just a new chapter for Blackbox but also an opportunity for a whole new story. 14 How AI will reinvent the market research industry, Qualtrics Experience Management Report, Qualtrics, Aug 2018, https://www.qualtrics.com/m/assets/wp-content/uploads/2018/08/AI-in-MR-Final.pdf , accessed Feb 2021. 8/9 This document is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Nov 2023. Copying or posting is an infringement of copyright. Permissions@hbsp.harvard.edu or 617.783.7860 EXHIBIT 1: NATURAL CONVERSATION FRAMEWORK (NCF) Natural Conversation Framework (NCF), developed at IBM Research, is a systematic framework for designing interfaces that work like natural conversation. "Conversational" means a natural-language interface that both recognises common conversational actions and preserves the sequential context of previous turns across future turns, so the agent can respond appropriately. The NCF provides a pattern language of generic, reusable conversational user experience (UX) patterns that are independent of any particular technology platform. The patterns are simplified forms of natural human conversation patterns documented in the Conversation Analysis (CA) literature, for example, those of sequence organisation or repair. The NCF consists of four main components: 1) an interaction model of "expandable sequences," 2) a corresponding content format, 3) a pattern language with 100 generic UX patterns and 4) a navigation method of six basic user actions. Source: Robert J. Moore, Raphael Arar, Conversational UX Design: A Practitioner's Guide to the Natural Conversation Framework, Association for Computing Machinery New York, April 2019, pp. 64 - 65, https://doi.org/10.1145/3304087, accessed Feb 2021. EXHIBIT 2: INITIAL IDEA OF THE CONVERSATION 9/9 This document is authorized for educator review use only by Saadat Alhashmi, Other (University not listed) until Nov 2023. Permissions@hbsp.harvard.edu or 617.783.7860 What the complexity in blackbox work process? and how the development chatbot changed it? Discuss all of them A) Data B) Design C) Strategy 2) From the strategic point of view, how would this development bb develop differentiatio among other competitors? market segmentation. 3) Given multiple success stories in the case? what lessons can be drawn to ensure the success of BB chatbot? 4) what are the barriers faced by the project team in the design process of Al ChatBot Choose one of them. Technical Strategical Ethical Customers must like the design. 1. Explain the complexity of the black box work process? and how the development of the chatbot changed it. 2. From the strategic point of view, how will this development help BB develop differentiation amongst its competitors? 3. Given multiple success stories in the case? what lessons can be drawn to ensure the success of the BB chatbot? 4. what are the barriers faced by the project team in the design process of AI ChatBot? Choose one of them. 5. What were the research and design methodologies deployed in the prototyping stage