VoiceOps-fying Low-Latency Intelligence Extraction from Messy Audio Streams — Dippu Kumar Singh

Talk transcript

1. 0:00

   Right. Hello, everyone. Welcome to the AI Engineer 2026 Online Track. Um, good morning, good afternoon, good evening, depending upon your time zone. My name is Dippu Singh, and I lead the initiatives in the emerging data technologies and AI architecture at Fujitsu North America.

2. 0:19

   Um, and today I will be, uh, giving you or maybe provide you a deep dive i-into a very specific but a very highly impactful engineering challenge, uh, which we have encountered in our, uh, contact center.

3. 0:34

   So with that, let me just quickly share my screen and get this going.

4. 0:49

   All right. So, um, um, this is the topic of the discussion, VoiceOps: Affine low-latency intelligence extraction from, uh, messy audio streams. Uh, when we talk about, you know, generative AI, we often, you know, focus on, uh, clean text inputs.

5. 1:11

   But if you go into the real world, specifically in the customer service or, uh, or maybe contact centers specifically, um, the data does not start as a clean text, right?

6. 1:24

   It starts as messy, quite overlapping, um, sometimes emotionally charged and, and it, it may have multi-channel audio streams also within that, right? And, and today we will explore the technical architecture which is required to capture that audio, process it with ultra-low latency, and use the generative AI to extract

7. 1:49

   structured, actionable, uh, business intelligence out of it. So with that, let's get s- started with this.

8. 1:59

   So here is our roadmap for the next twenty-five minutes. Um, first we will stage the, um, um, stage with what the current challenges are. Uh, we have to understand the operational realities and the intense human bottlenecks in, in the modern contact centers to understand why this engineering matters.

9. 2:24

   Uh, second, we will walk through the solution which is provided. I will break down our high-level architecture into four key components, um, detailing how we move from raw audio to a structured JSON using advanced summarization workflows.

10. 2:44

    Thirdly, we will look at the key outcomes, specifically like how these technical implementations, they translate into hard ROI and operational impacts. And finally, we will discuss the, um, roadmap ahead and, and being transparent about the current tech, uh, engineering constraints which we face, uh, now and then, and where we are taking this technology a-and the

11. 3:08

    roadmap ahead. So to engineer a great solution, we first need to deeply understand the problem. So let's look at the current state of the contact center operations.

12. 3:26

    So contact centers, they are the front lines of the customer experiences, but structurally they are breaking under the pressure. If you look at the industry data, um, over fifty percent of the contact centers, they identify, uh, hiring, training, and productivity as their most critical barriers.

13. 3:49

    Why? Because the job is incredibly difficult. When we analyze the reasons why the operators, they leave for the other professions, high stress is the number one factor as part of it.

14. 4:03

    Now, operators are expected to handle complex customer emotions, navigate the, um, multiple, uh, customer data platforms or the CRM systems involved into it and, and also document everything perfectly.

15. 4:20

    So if you think about this, this leads to a massive retention problem, right? Uh, we are caught in a negative spiral, um, where understaffing leads to higher stress for the remaining operators, um, which leads to, uh, high turnover and, uh, that ultimately implies to more understaffing, right?

16. 4:44

    So to break this cycle, we just, um, cannot hire more people. We have to fundamentally engineer the stress out of the workflow.

17. 5:00

    And the most glaring inefficiency in this workflow is something called, um, after-call workflow or ACW, uh, which we are going to discuss as part of this slide. Um, according to our baseline studies, the average contact center call, it typically l-lasts about like six point five minutes.

18. 5:24

    However, the average post-processing time where the operator types up the notes, summarizes the call, and selects the disposition codes, it, it takes like almost six point three minutes. So this is nearly like one is to one ratio we are talking about.

19. 5:40

    Um, and, and this also implies operators are spending almost as much time doing administrative data entry as they are actually talking to the customers. Um, furthermore, because the summarization relies on an individual operator's memory and the writing skills, the data quality is highly inconsistent.

20. 6:02

    Um, our core engineering mission here was clear, like use AI to target the after-call work, which is ACW. Um, if we can mechanize the summarization and the data extraction, uh, theoretically, we can reduce the post-processing time, um, almost by fifty percent or even more.

21. 6:26

    And, and this shifts the en-enterprise, you know, focus from merely just handling the calls to actually, you know, analyzing the voice of the customers for the business growth which we are looking out every now and then.

22. 6:43

    So how do we engineer that shift? Let's deep dive into the technical solution and the architecture we built to solve this. Um,

23. 6:58

    so we designed a four-stage low-latency pipeline to transform the conversational audio into a structured business intelligence, um, with minimal human intervention. And it starts with voice capture, which is tapping into the telephony system to extract raw, high-fidelity audio streams.

24. 7:21

    And that flows into our speech-to-text engine, STT, uh, which is responsible for high-accuracy transcription. Um, next is the brain of the system, the generative AI core. This is where we do the heavy lifting of the intent recognition and summarization.

25. 7:41

    And finally, the customer data sync layer, which translates those AI insights into API calls to update the, uh, uh, customer data or CRM data automatically. Uh, let's look at the engineering under the hood for each of these components in detail.

26. 8:04

    So the first component is the voice capture. In AI, um, the typical rule is garbage in equals garbage out. So if your audio intake is flawed, the LLM will hallucinate later on.

27. 8:19

    Um, we do real-time, uh, audio intake, so applying noise filters to strip out the, uh, back office chatter or, um, you know, anything of those sort, which is creating an attenuation is important and normalize the audio level is very, very, very important for us.

28. 8:39

    Crucially, if, if we perform these channel mapping, uh, we absolutely must espit- split the, uh, stereo audio to isolate the agent on one channel, say, on the very left, and the customer on the other side, which is the very right, right?

29. 8:55

    So if, if you mix them into a single mono track, um, like kind of overlapping with each other, the AI will struggle to figure out who said what and, and thereby, you know, ruining the entire downstream summary.

30. 9:11

    So, um, uh, it's important that we have that channel mapping intact, and it separates which-- who is what and who is saying what, right? Um, finally, we apply a security layer because sometimes the audio streams can contain the credit card numbers or maybe passwords or any-anything which is personally identifiable information, PII.

31. 9:35

    So we utilize buffer management and early-stage PII, uh, masking technique, uh, so that the sensitive data, um, it never hits the LLM memory banks whenever we are moving ahead in the channel.

32. 9:53

    Now next, the audio, it hits the, um, speech-to-text engine. For generative AI or the LLMs to summarize the, um, data effectively or your response effectively, we found that the speech-to-text, the STT, um, accuracy must be above ninety percent.

33. 10:15

    Uh, we utilize advanced acoustic modeling to map the, um, phonemes and filter out any regional dialects. We then apply the language logic utilizing, like, domain-specific dictionaries.

34. 10:33

    Uh, just for example, if it's an insurance agent, the, um, speech-to-text engine, STT, needs to know the difference between a term life and a turn right. Both of them are very close to each other, but still there should be a difference between them.

35. 10:51

    Uh, finally, post-processing is also vital. Uh, we use inverse text normalization and auto punctuation. Uh, for example, if, if a customer says five thousand dollars, the speech-to-text, um, engine, it should must output that into numerical fashion.

36. 11:09

    And this numerical formatting, it, it drastically improves the LLM's ability to extract the entities in later point in time.

37. 11:21

    Now we reach the generative AI core. We are not just throwing a raw trask-- uh, raw tras-- uh, transcript at, uh, LLM, but we are also asking it to summarize it.

38. 11:33

    Like, we use a highly orchestrated approach in this case. Um, in our orchestration layer, we use specific prompt templates. Our, um, setup showed that if, if you just ask an LLM to summarize a call, it outputs a messy narrative paragraph.

39. 11:53

    So instead, we use few-shot libraries to instruct the LLM to output separate bullet points. Um, one list for customer inquiry, uh, and a separate list for operator's action. Uh, then comes the reasoning layer.

40. 12:09

    In the reasoning layer, we extract the intent. We provide the LLM with a predefined list of customer, uh, call reasons like, uh, cancellation or new application or any kind of claim status, and instruct it to classify the transcript and output the reason why it choose that specific classification.

41. 12:34

    And finally, the trust layer, where we apply the token optimization to keep the, um, latency low and runs the, uh, automated hallucination checks to ensure the, uh, generated summary is strict, um, and it is grounded in the transcript.

42. 12:58

    Now, the final technical hurdle is getting this beautiful data back into hands of the business. Our API gateway acts as a schema mapper. It takes the JSON output from the LLM, uh, and maps the field like customer intent or resolution status and, and directly to

43. 13:23

    some corresponding fields i-in the, um, um, uh, based on the company's CRM system or any customer data which we have laid out via some REST APIs. So we don't remove the human entirely in this.

44. 13:36

    We use a verification step in between. Uh, the operator, it sees the AI-generated summary auto-populated on their screen. Uh, they do a quick visual field validation, make some minor edits if necessary, and then just click the Confirm.

45. 13:53

    Uh, simultaneously, this structured data, it flows into our business intelligence models, um, aggregating the voice of the customer data for management dashboards and automatically, you know, flagging the candidates for new FAQ, um, data entries.

46. 14:17

    Now, to tie the architecture together, this is the linear workflow logic of the data pipeline. We take the raw transcript complete, um, with the time indexing, uh, confidence scoring and denoising.

47. 14:33

    We pass this-- We pass it through the s-speaker separation. Because we split the stereo channels, um, in the step one, um, we can easily stitch the dialogue together logically, like customer said X, agent said Y.

48. 14:51

    Uh, then we move to the context deduct- deduction, where the LLM spots the entities like account numbers, um, or product name or customer name, run the sentiment analysis and recognizes the intent.

49. 15:07

    And the final state is the structured output. So instead of a wall of text, uh, the system, it outputs like clear, uh, and a clean JSON schema, matching some predefined, uh, customer data or maybe CRM templates which we have i-in the enterprise, and then categorized neatly into bullet points.

50. 15:28

    And, and this strict formatting is what turns an unstructured conversation into a database-ready asset.

51. 15:40

    So what happens when we deploy this architecture in a real contact center environment? Let's look at the outcomes.

52. 15:50

    The operational impacts out of the implementation were quite immediate and, and highly measurable. Um, look at the, um, ACW time, um, under the manual operation. After-- Average after-call work was six point three minutes.

53. 16:09

    Uh, powered by our AI workflow, that dropped to like three point one minutes, which is like almost fifty percent reduction in the processing time. Um, if you calculate that across five hundred seats, handling thousands of calls a day, you are looking at a massive operational saving, the equivalent of almost reclaiming dozens of full-time headcounts purely

54. 16:34

    from efficiency standpoint. Um, the next aspect is of data entry quality. It, it moved from a highly subjective and a variable to highly standardized and uniform output. Uh, the inquiry categorization or the call reason tagging, uh, it moved from being dependent on an operator mode or memory,

55. 16:59

    um, to, um, being strictly logic-based, resulting in a highly consistent voice of the customer data set for the, uh, management. And ultimately, by removing this repetitive administrative burden of typing out notes, we reduce the cognitive load on the operators, thereby stabilizing the operations and directly combating the stress which is linked

56. 17:24

    with the staff. And, and this ultimately reduced the turnover we identified, um, at the very beginning of our discussion.

57. 17:35

    While the results were fantastic, the, the engineering work is never done. Um, and, and let's talk about the constraints we face, uh, and our roadmap for the future.

58. 17:50

    Um, we are currently navigating three main constraints. First is the, um, source t- source-to-text accuracy, STT accuracy. The entire generative AI summary, it relies on the transcript.

59. 18:08

    So if the STT engine Uh, fails to pick up heavy accents or poor audio quality. The LLM has nothing to work with, so the STT optimization is a continuous battle for us.

60. 18:22

    Uh, second is the initial setup cost. While the long-term ROI is massive, the initial consumption of the API tokens, especially running complex LLM reasonings on long twenty-minute, uh, transcripts can be costly and, uh, especially during the initial scaling phases, those are tough ones.

61. 18:44

    And, and we are constantly working on token optimization techniques to bring this number down. Uh, the third is the security and compliance. Um, handling PII, uh, per, uh, any kind of sensitive information in audio streams is very complex.

62. 19:03

    Um, and ensuring robust masking before the hit-- data hits the cloud endpoint is a strict requirement. Um, and, uh, because of this, we add some layers and it ultimately reduce-- increases the latency and also adds some overhead from architectural standpoint.

63. 19:23

    So we are still figuring it out, how we can reduce those extra layers and make it much more robust component-wise.

64. 19:34

    All right, so to address these constraints, um,

65. 19:39

    a-and, and push the boundaries of what's possible, our roadmap is currently broken down into three phases. Uh, phase one, it focuses on explainable AI. Um, we want to move beyond just summarizing calls to actually, you know, coaching the operators.

66. 19:59

    We are engineering the systems to analyze the audio post-call and provide operators with instant private feedback on their, uh, either soft skills or empathy level or any kind of accuracy from a information standpoint.

67. 20:18

    The phase two, it targets the predictive staffing by taking the massive amount of categorized intent data we are now capturing. We can feed sa-- the same exact data, intent data into a time series analytics.

68. 20:35

    Um, and, and this will allow the workforce management to accurately forecast the call volumes spikes based on those specific topics and thereby optimizing the shift scheduling. Uh, uh, the phase three is perhaps the most important for human well-being, uh, combating the customer harassment.

69. 20:56

    Uh, uh, contact center agents, they mostly face an increasingly amount of verbal abuse. Um, and we are developing, uh, a low-latency sentiment and acoustic analysis that can detect, um, when a customer becomes abusive.

70. 21:14

    Um, ultimately, the system can, you know, alt-- trigger some triggers, uh, I mean, an alerts or something which is important from notification standpoint to a supervisor or anyone who is there in the-- from the management, upper management standpoint, um, or, or maybe seamlessly transfer the call to an AI voice agent just to protect the human operator, uh,

71. 21:39

    mental health in, in case of these tough conversations.

72. 21:46

    All right. Um, so, uh, by applying these rigorous engineering, uh, techniques to the messy audio data, we can definitely transform the contact centers from, uh, call centers of like high stress into a highly efficient and intelligence-gathering engines,

73. 22:12

    uh, that protect their workforces. That's the whole idea of having this, right? Um, thank you so much for your time, uh, and listening to me. Um, I have included my QR code, which I will just flash in here, um, so that you can grab me over LinkedIn.

74. 22:32

    Uh, and please feel free to connect if, if you would like to discuss, uh, the architecture or any prompt engineering strategies related to the discussion which we had in here, and happy to connect and, um, make things, uh, work things out between us, and we can have more conversations.

75. 22:50

    So thank you so much for listening to me and have a good one. Bye.