Building efficient hybrid context query for LLM grounding

Talk transcript

1. 0:00

   [on-hold electronic music] Hey, hey, hey.

2. 0:18

   How's everyone? This is Simrat Hanspal, technical evangelist at Hasura, and today I'm going to talk to you about building efficient hybrid RAG queries.

3. 0:29

   Let us understand this with the use case of product search in e-commerce domain. Present day, product search is mostly keyword-based. Keywords are not great at capturing the complete intent of the user's search query, so you want to move to using natural language.

4. 0:45

   But product search can be either contextual, where you're looking for-- where you're searching for product based on the descriptive nature, or, or it can be completely structured, where you're querying based on the structured fields, or it can be both.

5. 0:58

   Large language models are great, but they're frozen in time, and they cannot solve task on data they have not seen before. One of the ways to expose the unseen data to large language model is by providing context to the question alongside the question.

6. 1:13

   This helps the large language model generate more accurate and grounded answers. This powerful technique is called Retrieval-Augmented Generation, or RAG in short. So you see, we need to build a RAG pipeline for our product search use case.

7. 1:28

   We also need to make sure that our RAG pipeline is production-ready and will not leak any sensitive data, even if prompted. This security concern has been one of the primary concerns of enterprises when building gen AI applications.

8. 1:45

   Data-driven applications have been around for a while. Then why are we talking about secure data retrieval all over again for gen AI applications? Well, this is because we are seeing a paradigm shift in application development.

9. 1:59

   With data-driven applications, data is mostly constant, and it is the application or the software that evolves for any different or new functionality. For example, product search on current e-commerce websites would pick constant data feeds.

10. 2:15

    Only the records or the results would change.

11. 2:20

    While in context, um, context-driven or RAG application, the data is no longer a constant data packet, and it needs to adapt to the dynamic needs of the user's natural language query.

12. 2:33

    With natural language query, there is no structural limitations, and it can-- and it l- gives a scope for malicious attack.

13. 2:43

    Good news, Hasura enables you to build secure data API over your multiple different data sources in no time. Hasura APIs are GraphQL APIs, and hence they're dynamic in nature, so you get unified, dynamic, secure data API in no time.

14. 3:00

    Just what we needed. So let's get started with building a RAG pipeline for our product search use case.

15. 3:08

    Let us again look at what are the different queries that we, uh, that we can expect for our RAG applications. We can have semantic search, where we are searching based on semantic similarity with product description from product, uh, uh, vector DB.

16. 3:24

    We can also have structured search, where we are searching based on structured fields in the relation database, um, like for example, price and category in Postgres, and this requires converting the natural language query into a structured query like SQL or GraphQL.

17. 3:42

    Then we can also have hybrid queries. These searches have the elements of both semantic and structured queries. With Hasura, we don't need to build separate data APIs for each of them.

18. 3:53

    We can build a unified data API for all three of them.

19. 3:58

    So let's get started. We start by connecting our multiple different data sources with Hasura, and then we query it using a single GraphQL API. I've also built a Streamlit application which takes in the user input, calls the large language model, generates a GraphQL AP-- query, which then gets executed on Hasura.

20. 4:20

    So let's head over to Hasura Console to get a feel of what it looks like. To start, we'll go to the Data tab to connect all of our different data sources.

21. 4:31

    I'm not gonna do that because I have my, um, product Postgres table and product vector table already integrated. As I mentioned before, you can use Hasura to query both, um, your relational and vector DB and multiple data sources using a single GraphQL API.

22. 4:50

    But for the sake of simplicity of this demo, I'm gonna be using only the vector DB. So I'm using Weaviate in this case, where I have my vectors, and I have also got my price and category structured fields here.

23. 5:05

    One thing to note here is that I have used Hasura's event to auto-vectorize, um, my records into my vector DB, which means as and when a new record got inserted into my Postgres table, it got auto-vectorized and saved in my vector DB.

24. 5:21

    Amazing, I know. So let's go back to... Let's go to our API tab. This is where you will-- you can, uh, play around, execute different queries, and see the results.

25. 5:35

    Nice. Now that we have gotten a fairly decent sense of what Hasura Console is like, we can move to the Streamlit app that I have created. As you can see, there are a few configurations on the left-hand side panel.

26. 5:50

    So you have Hasura's endpoint and admin secret. This is required to connect with Hasura securely. And then I also have OpenAI's, um, API key. This is required for the Chat Completion API that I'm using.

27. 6:05

    So let's begin. Let's, uh, begin with querying the three different, uh, context, uh, that we were talking about that we want to fetch. So let's start with purely semantic one.

28. 6:17

    Let's look at the different product descriptions that we have and pick something. Let us pick, uh, products on essential oils. So let me say, "Show me essential oils for relaxation."

29. 6:41

    Great. So we've gotten the GraphQL query, which has identified essential oils for relaxation as the descriptive part of the query, which we want to find in our vector DB by doing a semantic search, and we can also see that we have gotten the results for this query.

30. 7:01

    Nice. Let's go over and execute a structured query. Price is a good field to execute, execute a structured query. So let's say, um, let's say, "Show me all products where less than price

31. 7:19

    five hundred dollars." Great. So it has rightly identified that there is a price filter with the less than condition,

32. 7:33

    and it shows you all the different products with price less than five hundred. Nice. Let's execute a hybrid query now.

33. 7:47

    Let's say looking for essential oil diffusers in the price range of five hundred to thousand dollars.

34. 7:58

    Nice. So we got a GraphQL query where it identified amazing essential oil diffuser as the semantic search query, and then the price filter, which is between five hundred to thousand, and we received our results.

35. 8:15

    Nice. So far, we have executed only the happy flows. Um, we have not looked at any other query where of unhappy flows. But let's say I had an evil intent, and I wanted to execute a malicious query, uh, which is not the typical queries that we just looked at.

36. 8:35

    So I have a malicious query. Let's execute this. So this one is requesting to insert a product of hair- hair oil product, um, with the name special oil and

37. 8:51

    price of [REDACTED:generic_id] [REDACTED:generic_id] dollars. Category is home. Fantastic hair oil is the description, [REDACTED:generic_id] let's also add the product ID [REDACTED:generic_id] say this is [REDACTED:generic_id]. Okay, let's execute this.

38. 9:10

    So as you can see, it, it has generated a GraphQL query of type insert mutation. But what we see is that it has also inserted the query. So let's go back to our table and console and look for product ID equal to

39. 9:31

    [REDACTED:generic_id]. Let's just remove the quotes because this is

40. 9:38

    our integer field. There you go. We have the product which has gotten inserted into the database. Um, this was not the intended behavior. This is not what should have happened.

41. 9:51

    So let us quickly go back to our Hasura console again, and this time, we are gonna be defining a new role with very restricted permissions so that we only provide select permission and such that this does not happen again.

42. 10:05

    So I'm gonna create a new role. Let's call it Product Search Bot, and I'm gonna provide only search permission. Let's go without any checks. I'm gonna keep it really simple.

43. 10:18

    Let me allow all the product, all the columns to be accessible for this role.

44. 10:24

    That's about it. Nice. So the role has gotten inserted. Now let's query. Same thing with the new role, so let's say product search bot. But this time, let me just modify this query a little bit and say seven thousand and two.

45. 10:42

    Okay. So let's execute this and see what happens.

46. 10:49

    Nice. So we got the same insert mutation query, um, to be generated, but this time there was an edit executing this rightly so because we have defined a role which does not have the permission for insert queries.

47. 11:04

    Great. So this is all from me. Thank you, everyone. Um, thank you once again. So let us really quickly recap. In this demo, we learned how we can use Hasura to build hybrid query context, um, for your sophisticated RAG applications like product search.

48. 11:21

    If you like the demo or would like to use Hasura for your RAG application, please reach out to me. These are my contact details, and thank you so much. [upbeat music]