Beyond Prompt Engineering – Extending OpenAI with your data, systems, and API’s

At the moment it seems like every second LinkedIn or post is about prompt engineering (‘sixteen secret prompts to improve your ChatGPT’). There is no doubt that prompt engineering is a critical part of the process when using ChatGPT or Open AI. These prompts help guide the AI engine by clarifying your intentions and how to format and style a response. However, prompt engineering does not really provide OpenAI with any significant amounts of data that is not already part of its pretrained model.

For many scenarios giving OpenAI the ability to query internal databases, applications, APIs, in real time and then using those results as part of a generative AI response can be a game changer. This can vastly improve employee efficiency in a host of ways and surface valuable yet latent data for use in new and innovate ways.

Let’s work through a specific use case and how it can be done.

If I ask the general public ChatGPT engine about company specific data, I unsurprisingly receive the following:



However, what if the response was more like this?



(With appropriate authentication checks and balances of course) 

Most organisations will have a range of internal systems and typically there is a different application or dataset for each of them:


Each of these systems will have their own UI, website, or application.


Each of your specialised systems should have these types of reports built in, however:

  • You probably need a special application or website for each one.
  • It may take several minutes to log on, click around and generate the data.
  • Layouts and export formats are often fixed.

Could you generate these reports in 5 seconds?  From a single location?

Now let’s get a bit more complex.  Can you query across all these applications and systems?

Could you answer:

  • Do the people who have completed the ‘Ninja Sales’ course have higher sales?
  • How does staff tenure affect sales attainment?  Are the junior staff struggling?
  • Are people who have not taken leave for 12 months less effective?

To do this you would probably:

  • Run reports in each system.
  • Export to Excel or similar
  • Perform vlookups or similar and try and match fields.

You could also build/commission a custom report (e.g. PowerBI):

  • This is great for enterprise reports, but may not be cost effective for adhoc queries.
  • You need to know what to ask for.
  • Reports then need to be maintained.

So – How can Azure Open AI help?

Open AI supports an extension mechanism known as tools (previously called functions).

You describe to Open AI what the tool is, what it does and the data it needs. This description is in almost a narrative form:



With this information, OpenAI then ‘knows’ how to look up custom databases and APIs and get the info needed:



The language model itself does a fantastic job of working out what tools are needed, the order in which they are needed and the data that needs to be passed in and out. Open AI then takes the real-time data from these tools and combines into a generative AI response.

Lets look at something more complex:


To answer this, the following had to occur:

  • OpenAI had to understand what I was asking and assess my request against the tools it had available.
  • A call was made to get_TeamDetails to return a list of all members of the sales team.
  • Then for each person in the sales team:
    • A call was made to get_SalesDetails to look up their January sales results AND
    • A call was made to get_TrainingDetails to check what training was completed.
  • OpenAI then determined who was the highest performer (from context alone) and that they had not completed the Ninja Sales course.
  • The results were all combined into a generative AI response.

Notes :

  • This took about 5 seconds to execute and display results.
  • Open AI was not told how to go about solving this problem. All it was told was the functions available and the data they returned.   

You can even ask OpenAI how it would go about solving it: 


How do I make my data and systems available to the OpenAI tools?

There are a number of ways to do this, and it depends on whether the data is in a database (e.g. SQL or Oracle) exposed via an API or some other system.

One of the easiest ways is to use Azure Logic Apps. You can simply wrap the data source and make it available as a HTTP API with only a couple of steps.


Using LogicApps (part of Azure Integration Services) in this way enables nearly any database, API or source to be quickly made available without needing any modifications.


Iterative querying

Are the results always perfect? No

Does the LLM always understand my meaning on the first attempt? No

But because it is a conversation, data can be enhanced as you go :


Want to know more?

• Visit us us our booth at the Microsoft AI Tour in Sydney on February 7th 2024 for live demos and whiteboarding. 

• Attend the breakout keynote on this topic at the event.

Contact us for more information.


Contact Us

Receive a FREE 30-minute consultation with one of our BI Consultants

Find Out