Building a chatbot is incredibly simple thanks to LLama index and Open AI. Here I go over creating a gRPC service in python that utilizes LLama Index to make requests to Open AI. We will finally containerize the service.

When building a chatbot with OpenAI, the first difficult problem you encounter is deciding on the format for the response. You will need to decide whether to stream the response and how the LLM will format the response. Then you will have to deal with parsing and rendering the response in the client. Here, I discuss why you should stream the response, have the LLM return the response in markdown and use the marked library to parse the response.