Speaker 1: Welcome to the Deep Dive. We're the show that helps
Speaker 1: you cut through all that information noise and really get
Speaker 1: to the insights you actually need. If you've ever felt
Speaker 1: like you're trying to well drink from a fire hose
Speaker 1: when you're looking at the world of AI, especially genitive AI,
Speaker 1: you are definitely not alone. It's a lot. So today
Speaker 1: we're taking a deep dive into something really practical, unlocking
Speaker 1: creativity with Azure open Ai. It's basically a guide to
Speaker 1: using these really advanced AI models effectively. Our mission here
Speaker 1: is simple, cut through the complexity, pull out the most
Speaker 1: important stuff, the surprising facts, so you can get up
Speaker 1: to speed fast on how these tools work and importantly,
Speaker 1: how they're being used out there in the real world.
Speaker 1: Think of this as your shortcut, you know, to understanding
Speaker 1: the what, the how, and the why it matters. For
Speaker 1: Azure open Ai, the source for dipping into is super comprehensive.
Speaker 1: It goes from the absolute basics right through to advanced
Speaker 1: stuff security, how to actually run these things, the whole
Speaker 1: nine yards. Okay, so let's unpack this a bit. When
Speaker 1: we talk about large language models lllms, what are we
Speaker 1: actually talking about, Like, what's the core idea.
Speaker 2: So at their very core, lms are about taking human language,
Speaker 2: our text, and turning it into something computers can genuinely
Speaker 2: work with, not just store. It starts by breaking the
Speaker 2: text down into what are called tokens. These are usually
Speaker 2: words or sometimes parts of words.
Speaker 1: Okay, tokens, got it?
Speaker 2: Yeah, And then these tokens get converted into something called embeddings.
Speaker 2: These are numerical vectors, basically long strings of numbers. You
Speaker 2: can sort of imagine these embeddings like a really sophisticated map,
Speaker 2: where the position of each point tells you the meaning
Speaker 2: of that word or phrase and how it relates to others.
Speaker 1: Ah. So it's not just the word itself, but it's
Speaker 1: meaning in context exactly.
Speaker 2: That's how the computer starts to grasp the nuances the context,
Speaker 2: not just isolated words. Now, the real breakthrough tech here
Speaker 2: is the transformer architecture. Older models they really struggle to
Speaker 2: keep track of context in long pieces of text. They'd
Speaker 2: sort of forget the beginning, right.
Speaker 1: I remember that limitation.
Speaker 2: Yeah, But the transformer, with its self attention mechanism, totally
Speaker 2: change the game. It lets the model way how important
Speaker 2: different words are to each other, even across a really
Speaker 2: long sequence. It captures those deep relationships. And when an
Speaker 2: LM actually generates text, it does it word by word.
Speaker 2: It's called auto regressive generation.
Speaker 1: Auto regressive Okay, think of.
Speaker 2: It like a game where each move, each word is
Speaker 2: based on all the previous ones. It helps maintain context
Speaker 2: and coherence, even for really complex ideas. And these models
Speaker 2: are just massive, huge. They run on big clusters of
Speaker 2: computers and usually access them as a service through an
Speaker 2: API because they've been trained on just enormous amounts of text.
Speaker 1: Data like a skilled improv artist building on what came before.
Speaker 1: That's a helpful analogy. So okay, that's the foundation. Then
Speaker 1: what about foundation models? What makes them special?
Speaker 2: Well, what's really interesting about foundation models is while their
Speaker 2: main job is basically predicting the next word, their sheer
Speaker 2: scale changes things. They're trained on these immense diverse data sets.
Speaker 2: We're talking terabytes of data often, and this training gives
Speaker 2: them what are called emergent capability.
Speaker 1: Emergent capabilities meaning.
Speaker 2: Meaning they can do a whole bunch of tasks they
Speaker 2: weren't specifically programmed or trained for, often really well, sometimes
Speaker 2: just needing a few examples or even none. The main
Speaker 2: advantages are well. First that performance. It leads to really
Speaker 2: big productivity games. Think of them like a super efficient
Speaker 2: assistant for tasks that usually take a lot of time
Speaker 2: customer service processing data. They can speed things up dramatically.
Speaker 1: A turbo charger for the team.
Speaker 2: Yeah, pretty much, but this is important. They have limitations.
Speaker 2: The big one is hallucination.
Speaker 1: Ah heard about this.
Speaker 2: It's when the LM generates stuff that sounds totally plausible,
Speaker 2: really confident, but it's just not factually accurate or maybe
Speaker 2: even completely made up.
Speaker 1: So it's not lying, just pattern matching gone wrong exactly.
Speaker 2: It's confidently producing text that fits a pattern even if
Speaker 2: reality doesn't match. That's why human oversight is absolutely crucial.
Speaker 2: We need to ground them, which we can talk about.
Speaker 2: Another limit is the context window. It's basically the model's
Speaker 2: short term memory, how much info it can juggle at once.
Speaker 2: You know, some big models like GPT four to oh
Speaker 2: can handle say one hundred and twenty eight thousand input tokens,
Speaker 2: which is huge, but there's still a limit. Feed it
Speaker 2: too much and it just can't process it all simultaneously.
Speaker 1: Okay, so potential for errors, memory limits, but still incredibly powerful.
Speaker 1: So where are we seeing these foundation models really making
Speaker 1: a difference in the real world despite those caveats.
Speaker 2: Well, the fleckibility is just amazing. It's touching almost every industry.
Speaker 2: In content creation, for example, they're not just writing generic stuff.
Speaker 2: They can generate targeted marketing copy, blog posts, social media updates. Yeah,
Speaker 2: speeding up content pipelines hugely faster content okay. And in
Speaker 2: customer support, handling tons of common questions automatically. That frees
Speaker 2: up human agents for the really tricky, nuanced problems. Beyond that,
Speaker 2: text summarization is a big one, getting the gist of
Speaker 2: long documents quickly, powering sophisticated chatbots, virtual assistance for personalized help,
Speaker 2: even creative writing assistance, you know, brainstorming plots or dialogue.
Speaker 1: Interesting. What about more specialized fields.
Speaker 2: Yeah, definitely making inroads in healthcare for instance, they can
Speaker 2: help analyze initial patient info maybe symptoms alongside some images.
Speaker 2: But and this is critical, they are not built to
Speaker 2: interpret specialized medical scans or give medical advice that needs.
Speaker 1: A professional, very important distinction.
Speaker 2: Absolutely. Yeah, And we also see them in cybersecurity for
Speaker 2: analyzing potential threats and language learning apps creating accessibility tools
Speaker 2: like audio descriptions for videos. The list just keeps growing.
Speaker 1: Wow, that's a massive range from marketing copy to analyzing
Speaker 1: medical info sort of. Okay, now let's pivot. This is
Speaker 1: where it gets really interesting for businesses. Right, how does
Speaker 1: Microsoft's Azure open AI fit in? We hear about this
Speaker 1: big partnership.
Speaker 2: You're right, that partnership is central. Azure OpenAI Service or AOAI,
Speaker 2: is Microsoft's way of bringing these powerful open AI models
Speaker 2: into the enterprise world, but with a heavy focus on security,
Speaker 2: compliance and and manageability. It gives you secure rest API
Speaker 2: access to all the big open AI models GPT four Turbo,
Speaker 2: the new GPT four h GPT four oh Mini, GPT
Speaker 2: three point five Turbo for text tasks, Whisper for audio,
Speaker 2: Daily three for images, and the embedding models.
Speaker 1: So the models everyone's talking about, but package for business exactly.
Speaker 2: But the key difference with Azure open Ai is the
Speaker 2: enterprise grade stuff that's only available on Azure. We're talking
Speaker 2: robust security controls, private networking options so your data doesn't
Speaker 2: touch the public Internet, meaning strict compliance standards, broad geographic availability,
Speaker 2: and really important built in responsible AI content filtering.
Speaker 1: Okay, those enterprise features sound critical. Can you quickly run
Speaker 1: through the main model types.
Speaker 2: Again, sure you've got the GPT four family that's the
Speaker 2: top tier, like GPT four Oh, GPT four O Mini
Speaker 2: and Turbo. They have advanced reasoning, big context windows. GPT
Speaker 2: four in takes one hundred and twenty eight thousand input tokens,
Speaker 2: which is huge, a whole book almost pretty much, and
Speaker 2: GPT four Mini is interesting because it can output a
Speaker 2: lot of tokens up to sixteen thousand, great for longer
Speaker 2: respons bonses. Then there's GBT three point five Turbo, often
Speaker 2: the go to for being capable, the cost effective, especially
Speaker 2: for chats, and of course Whisper for audio, Dally three
Speaker 2: for images, and the embedding models which are essential for
Speaker 2: any kind of smart search or understanding meaning.
Speaker 1: And who gets access? Can any business just sign.
Speaker 2: Up right now? Access is mostly for enterprise customers and partners.
Speaker 2: You typically apply using your company email. It's a deliberate
Speaker 2: approach really, Microsoft wants to ensure these powerful tools are
Speaker 2: deployed responsibly and securely in business settings with the right
Speaker 2: support in governance structures in place.
Speaker 1: Makes sense for managing something this powerful. Okay, let's go deeper. Now,
Speaker 1: some of the more advanced capabilities that really unlock new potential.
Speaker 1: Tell us about those embedding models in Azure open AI.
Speaker 1: What do they let you do?
Speaker 2: Right? Embeddings they are absolutely fundamental for what we call
Speaker 2: semantic understanding and similarity searches. Instead of just matching keywords
Speaker 2: like finding car when someone types car, embeddings capture the meaning.
Speaker 2: So if you search for fast car, it can find
Speaker 2: dot com U means talking about rapid automobiles because it
Speaker 2: understands those concepts are similar.
Speaker 1: Much smarter search then exactly.
Speaker 2: Much more relevant results. Now there are older versions like
Speaker 2: ad A zero zero two, but the newer ones text
Speaker 2: embedding three small and text embedding three large, are well.
Speaker 2: They're significantly better. Text ebedting three small is much more
Speaker 2: cost effective and shows big performance jumps, especially for multi
Speaker 2: lingual stuff. Text embedding three large is the top performer
Speaker 2: overall for accuracy.
Speaker 1: Better and cheaper. Nice.
Speaker 2: And here's a really cool thing about these new models,
Speaker 2: a real aha moment. They use something called Matryoshka representation
Speaker 2: learning am.
Speaker 1: Like the Russian dolls exactly.
Speaker 2: It means you can actually shorten the embeddings, literally chop
Speaker 2: off numbers from the end of the sequence without them
Speaker 2: losing their core meaning. This is huge because shorter embeddings
Speaker 2: mean less storage, faster searches, lower costs, often while keeping
Speaker 2: or even improving performance compared to older, longer embeddings. It's
Speaker 2: incredibly efficient.
Speaker 1: That's amazing, trimming the th without losing the substance. Yeah.
Speaker 1: So you create these smart embeddings, where do you put them?
Speaker 1: Why are Azure vector databases important here?
Speaker 2: Good question. You need a special kind of database optimized
Speaker 2: for storing and searching these high dimensional vectors. That's where
Speaker 2: Azure vector databases come in. Their whole point is to
Speaker 2: enable really fast, really precise similarity searches based on that
Speaker 2: semantic meaning we talked about, not just keyword matching, find
Speaker 2: related concepts instantly across huge data sets.
Speaker 1: And Azure has options for this.
Speaker 2: Oh yes, Azure ai search is a big one. Interestingly,
Speaker 2: open Ai actually uses Azure ai Search for vector capabilities
Speaker 2: in chat GPT itself.
Speaker 1: Wow.
Speaker 2: Yeah. And there's also Azure Cosmos dB with vector capabilities,
Speaker 2: Azure Managed Rettis, and even Postgres School with the PG
Speaker 2: vector expansion. Lots of choices depending on your needs, all
Speaker 2: designed for handling these complex numerical vectors.
Speaker 1: Okay, Earlier you mentioned that limitation hallucination where lllms can
Speaker 1: make things up. How does retrieval, augmented generation or RI
Speaker 1: help fix that?
Speaker 2: Right? Is direct answer to the hallucination problem. It works
Speaker 2: by grounding the model.
Speaker 1: Grounding it like keeping its feet on the ground pretty much.
Speaker 2: It connects the LM's internal knowledge with real world verified information,
Speaker 2: usually from an external source. Think of it like giving
Speaker 2: the model a factual reference library to check before it
Speaker 2: generates an answer, keeps it rooted in reality.
Speaker 1: How does that work in practice?
Speaker 2: So the process is quite elegant. A user asks a question. First,
Speaker 2: the system retree is relevant information from an external knowledge base,
Speaker 2: typically one of those vector databases we just discussed. Then
Speaker 2: the LM gets both the original question and this retrieved
Speaker 2: factual context. It uses both pieces to generate the final response.
Speaker 1: Ah, so it's using verified info to guide its answer precisely.
Speaker 2: The benefits are huge. Much better accuracy because it's using facts,
Speaker 2: richer context than just its training data, more flexibility because
Speaker 2: you can update the knowledge base without retraining the whole model,
Speaker 2: and it scales well.
Speaker 1: Sounds great. Are there downsides?
Speaker 2: Our challenges?
Speaker 1: Yeah?
Speaker 2: Getting the document segmentation right for the retrieval step is tricky.
Speaker 2: Making sure the retrieved info is genuinely relevant can be hard,
Speaker 2: and setting of the whole RMA pipeline could be complex
Speaker 2: and resource intensive.
Speaker 1: Okay, makes sense. Moving beyond just text, what about models
Speaker 1: that understand images too? Tell us about azure OpenAI is
Speaker 1: multimodal stuff, especially GBT four oh.
Speaker 2: Yeah, multimodal is a really exciting frontier. Models like GBT
Speaker 2: four oh can process and understand both text and images
Speaker 2: together in the same input, so.
Speaker 1: You can show it a picture and ask questions.
Speaker 2: About it exactly. This opens up tons of practical uses,
Speaker 2: automatically generating detailed captions for images, visual question answering asking
Speaker 2: what color is the car in this picture, content moderation
Speaker 2: for visual stuff in e commerce, maybe generating product descriptions
Speaker 2: just from photos, and as we touched on, even assisting
Speaker 2: with initial medical diagnostics by looking at symptoms and related images.
Speaker 2: But again with that crucial caveat, not for interpreting specialized
Speaker 2: scans or giving.
Speaker 1: It right always the caveat. Are there things that struggles
Speaker 1: with visually definitely limitations.
Speaker 2: It might not perform as well with non Latin alphabets
Speaker 2: and images, or very small or rotated text. Sometimes precise
Speaker 2: spatial reasoning like is the blue box exactly to the
Speaker 2: left of the red sphere? Can be tricky for.
Speaker 1: It still a massive leap. Now, how do these models
Speaker 1: actually do things in the real world interact with other systems?
Speaker 1: How does function calling work?
Speaker 2: Function calling is super interesting. The key thing to get
Speaker 2: is the model itself doesn't run the function.
Speaker 1: It doesn't, then what does it do?
Speaker 2: It intelligently figures out if an external tool or function
Speaker 2: is needed to answer the user's request. If it decides yes,
Speaker 2: it then generates the parameters or arguments that function needs.
Speaker 2: So the flow is like this model thinks a function
Speaker 2: call would help. The API response tells your application, Hey,
Speaker 2: call this function with these arguments.
Speaker 1: So my app does the actual work exactly.
Speaker 2: Your application takes those parameters, runs the function. Maybe it
Speaker 2: queries a database, calls another API, sends an email, whatever.
Speaker 2: Then your app sends the result of that function call
Speaker 2: back to the LM. The LEM then uses that real
Speaker 2: world result to formulate its final informed answer to the user.
Speaker 2: It's a really dynamic way to connect the AI to
Speaker 2: external systems.
Speaker 1: Got it? And building on that interaction idea, what's the
Speaker 1: assistance API? Sounds like you can build more complex agents.
Speaker 2: Precisely, the Azure Open AI Assistance API is designed specifically
Speaker 2: for building these more sophisticated stateful AI assistants, tailored to
Speaker 2: particular jobs. It comes with some really powerful built in tools.
Speaker 2: One is a code interpreter. This lets the assistant write
Speaker 2: and run Python code securely in a sandboxed environment.
Speaker 1: Python code what for all.
Speaker 2: Sorts of things, performing complex calculations, analyzing data directly from
Speaker 2: uploaded files like csvs, even generating charts or processing files.
Speaker 2: It's incredibly powerful for data tasks. Another key tool is
Speaker 2: file search. This allows the assistant to access and retrieve
Speaker 2: information from documents you provide to It.
Speaker 1: Ah like a private knowledge base for assistant exactly.
Speaker 2: It acts as an external knowledge source, letting the assistant
Speaker 2: answer questions using your specific, up to date information, going
Speaker 2: way beyond its original training. It uses vector embeddings under the.
Speaker 1: Hood for this, and function calling is part of this too.
Speaker 2: Yep, function calling is integrated right into the assistance API
Speaker 2: as well, so your assistant can use those external tools seamlessly.
Speaker 1: Okay, so assistance API for interactive smart agents. What if
Speaker 1: you just need to process a ton of stuff and
Speaker 1: you don't need instant answers like batch processing.
Speaker 2: That's exactly where the batch API comes in. It's designed
Speaker 2: for asynchronous, non real time processing jobs where you can
Speaker 2: wait a bit for the results. You basically bundle up
Speaker 2: a whole load of requests into a single file, submit it,
Speaker 2: and AZURE processes in bulk.
Speaker 1: What are the advantages of doing it that way?
Speaker 2: Two main things, costs and quota. You typically see a
Speaker 2: significant cost reduction, often around fifty percent compared to making
Speaker 2: all those calls individually to the standard real time endpoints.
Speaker 2: Plus you get a dedicated quota for backs processing separate
Speaker 2: from your interactive traffic. Azure guarantees completion within twenty four hours,
Speaker 2: though usually it's much much faster. Perfect for large scale
Speaker 2: content generation, data cleansing, summarization tasks.
Speaker 1: Things like that fifty percent cost reduction is pretty compelling. Okay,
Speaker 1: let's switch gears slightly. Fine tuning. This comes up a lot,
Speaker 1: but it raises a big question. When do you actually
Speaker 1: need to fine tune a model? Especially with powerful things
Speaker 1: like prompt engineering and r GAG.
Speaker 2: Available, That is a really critical strategic question. Fine tuning
Speaker 2: is different. It means taking an existing pre trained LLM
Speaker 2: and actually retraining it, adapting it using your own specific
Speaker 2: curated example data. It's a supervised learning process. You show
Speaker 2: the model examples given this input produce this exact output.
Speaker 2: You're teaching it a very specific behavior or style.
Speaker 1: Okay, so you're modifying the model itself. What are the benefits?
Speaker 2: Well, you can potentially get much higher quality responses for
Speaker 2: very specific niche tasks. You can effectively train it on
Speaker 2: more data than fits in this andar context window because
Speaker 2: the knowledge gets baked into the model weights, and sometimes
Speaker 2: it can lead to using fewer tokens in your prompts later,
Speaker 2: saving costs.
Speaker 1: So when is it the right call over just better
Speaker 1: prompting or RAG?
Speaker 2: You should really only consider fine tuning when prompt engineering
Speaker 2: in a RAG aren't getting you the consistent quality or
Speaker 2: accuracy you need for a specific problem. It's best when
Speaker 2: you have a unique domain or a very specific data
Speaker 2: set that's well prepared in high quality, and crucially, you
Speaker 2: need clear goals and ways to measure if the fine
Speaker 2: tuning actually work. Like quantitative metrics, how much data do
Speaker 2: you need? That's a key point. While technically you might
Speaker 2: start with just like ten examples to get any real benefit,
Speaker 2: to really shift the model's behavior. Usually need hundreds or
Speaker 2: more likely thousands of high quality examples.
Speaker 1: Thousands okay, that's a commitment.
Speaker 2: It is, and importantly, low quality or inconsistent examples can
Speaker 2: actually hurt the model's performance, making it worse, so data
Speaker 2: quality is paramount. The process involves preparing that data, carefully,
Speaker 2: running the training job, and then rigorously evaluating both safety
Speaker 2: and performance before deploying.
Speaker 1: Okay, let's circle back to interacting with the model. Prompt
Speaker 1: engineering you mentioned it's powerful. It feels like a real
Speaker 1: art form almost. It's not just asking a basic question, is.
Speaker 2: It not at all? It's absolutely critical. Prompt engineering is
Speaker 2: basically the art and science of crafting your input, your
Speaker 2: prompt to guide the LM towards the specific kind of
Speaker 2: output you want without changing the model itself. It's all
Speaker 2: about how you communicate your request to the AI. Think
Speaker 2: of a really good prompt as having several key ingredients.
Speaker 1: Ingredients like a recipe, kind.
Speaker 2: Of first unique context like imagine you're a travel agent.
Speaker 2: Then clear instructions, write a three day itinerary, add constraints
Speaker 2: focusing on budget friendly options. You might include variables or
Speaker 2: specific inputs. Mention the Eiffel Tower in the louver, specify
Speaker 2: the desired output format, provide the answer as a bulleted list,
Speaker 2: and maybe set the tone style in an enthusiasm, sick
Speaker 2: and friendly tone.
Speaker 1: Wow, Okay, that's quite detailed.
Speaker 2: It can be, And one really powerful element is providing
Speaker 2: examples or templates, like here's an example of a good
Speaker 2: itinerary item. Day one morning, visit Notre Dame cathedral, free entry.
Speaker 2: Now create the rest. Putting these elements together makes a
Speaker 2: huge difference in getting tailored, useful responses instead of something
Speaker 2: generic that.
Speaker 1: Makes total sense layering the instructions, What about more advanced
Speaker 1: strategies you mentioned guiding the AI's thought process.
Speaker 2: Yet beyond just the structure, there are strategies. Always aim
Speaker 2: for clear, unambiguous instructions. Asking the model to adopt a
Speaker 2: specific persona helps. Using delimiters like triple quotes or XML
Speaker 2: tags to separate instructions from content is good practice. Breaking
Speaker 2: down complex tasks into steps for the model is effective,
Speaker 2: and as I mentioned, providing examples is almost always beneficial.
Speaker 2: One really important strategy is often called give the model
Speaker 2: time to think.
Speaker 1: Time to think. It's not actually thinking though, right.
Speaker 2: Right, it's not conscious Yeah. Structuring the prompt to encourage
Speaker 2: a step by step process often leads to better accuracy
Speaker 2: on complex problems. Force it to outline its steps before
Speaker 2: giving the final answer. It's like asking your person to
Speaker 2: show their work in math reduces errors.
Speaker 1: Oh, okay, show you work. What about specific named techniques?
Speaker 2: So we have a kind of progression. Zero shot is
Speaker 2: asking a question cold with no examples. One shot gives
Speaker 2: one example, Few shot gives well a few examples. Adding
Speaker 2: examples dramatically improves accuracy by showing the model the pattern
Speaker 2: you want. Then there's chain of thought or code T.
Speaker 2: This is where you explicitly ask the model to explain
Speaker 2: its reasoning step by step before giving the final answer.
Speaker 2: It forces that show your work process and really helps
Speaker 2: with complex logic or math problems.
Speaker 1: So you see it's reasoning exactly.
Speaker 2: Building on that is tree of SATs or toe T.
Speaker 2: This is more advanced. It lets the model explore multiple
Speaker 2: different reasoning paths like branches of a tree, evaluate them,
Speaker 2: and then choose the best one. Great for complex planning
Speaker 2: or exploring possibilities.
Speaker 1: Oka more complex now, yeah.
Speaker 2: A couple more interesting ones. Program aided language model or
Speaker 2: pall MS. This is fascinating. The LM actually generates small
Speaker 2: snippets of code, often Python, to help it solve a problem.
Speaker 1: It writes code to help itself.
Speaker 2: Yes, like if you ask a complex math question, it
Speaker 2: might write and run Python code using an interpreter to
Speaker 2: get the exact numerical answer rather than trying to estimate
Speaker 2: it linguistically. Then there's react. This technique lets the model
Speaker 2: interleave reasoning steps with actions. You can decide it needs
Speaker 2: more information. Formulate a query to an external tool like
Speaker 2: a search engine or database. If you have function calling,
Speaker 2: get the result, and then incorporate that into its reasoning
Speaker 2: to continue.
Speaker 1: So it can actively seek out information.
Speaker 2: Yes, interact with tools to improve its response, and finally, reflection.
Speaker 2: This allows an agent to look back at its past
Speaker 2: actions and outcomes, receive feedback, often linguistic, and essentially learn
Speaker 2: from its mistakes to improve its strategy. Over time, it
Speaker 2: reflects on its own reasoning.
Speaker 1: Wow, okay, lots of powerful techniques there. So if you
Speaker 1: have all these prompting methods plus RG plus fine tuning,
Speaker 1: how do you decide where path to take? It seems complicated.
Speaker 2: It really boils down to a few key factors. Your
Speaker 2: specific goal. The resources you have and your team's expertise.
Speaker 2: Prompt engineering is almost always the starting point. It's low cost,
Speaker 2: fast accessible. You can often get great results just by
Speaker 2: crafting better prompts using those techniques.
Speaker 1: We discussed, Start simple, iterate exactly.
Speaker 2: If prompt engineering isn't enough, and especially if your application
Speaker 2: needs access to external changing information or needs to avoid
Speaker 2: hallucinations based on specific documents, that ra is usually the
Speaker 2: next step. It adds that grounding layer fine tuning is
Speaker 2: generally the last resort. It's more expensive, needs significant data
Speaker 2: and mL expertise. You'd only really go there if RAG
Speaker 2: and prompting aren't hitting the mark for a very specific,
Speaker 2: high value task where you need to deeply embed custom
Speaker 2: knowledge or style into the model itself.
Speaker 1: Okay, that clarifies the decision process. Now it's switching to
Speaker 1: deployment for any business using the security responsible use keeping
Speaker 1: things running smoothly, These are absolutely critical. How does Azure
Speaker 1: OpenAI handle that side of things?
Speaker 2: Microsoft puts a huge emphasis on this For compliance and
Speaker 2: data privacy, the Azure open Ai service meets strict standards
Speaker 2: like SOOC one, two and three. The really crucial privacy
Speaker 2: point is that the models are stateless They don't learn
Speaker 2: from or remember your interactions, your prompts, the completions generated,
Speaker 2: any embeddings, any data you use for fine tuning. None
Speaker 2: of it is shared with other customers. It's not sent
Speaker 2: to open AI. Microsoft doesn't use it to improve their
Speaker 2: base models as not shared with any third parties. Your
Speaker 2: data stays yours.
Speaker 1: That's a big reassurance for businesses. What about monitoring for
Speaker 1: bad stuff?
Speaker 2: Right? Microsoft does have real time abuse monitoring systems that
Speaker 2: scan for harmful content generation. However, and this is important,
Speaker 2: eligible customers can actually apply to opt out of this
Speaker 2: monitoring if approved, none of your prompt or completion data
Speaker 2: is stored for that purpose.
Speaker 1: Okay. Control over monitoring and content filtering?
Speaker 2: Yes, there's built in content filtering that runs alongside the models.
Speaker 2: It uses classification models to check prompts and outputs for
Speaker 2: categories like hate speech, sexual content, violence, and self harm.
Speaker 2: It operates on severity levels safe, low, medium, high. The
Speaker 2: default usually filters content rated medium or high severity. Businesses
Speaker 2: can request customizations like only filtering high severity or even
Speaker 2: disabling specific filters, but that typically requires justification and approval.
Speaker 1: What about securing access not putting API keys in code.
Speaker 2: Definitely best practice to avoid that. Azure uses manage identities.
Speaker 2: This lets your Azure services authenticate securely to Azure OpenAI
Speaker 2: without needing to embed keys directly in your application code.
Speaker 2: Much safer, and for network security, you can use private endpoints.
Speaker 2: This essentially connects the Azure OpenAI service directly to your
Speaker 2: private Azure network, disabling public Internet access entirely for that resource.
Speaker 2: All traffic stays within your secure boundary and data encryption absolutely.
Speaker 2: Data is encrypted both at rest and in transit. At rest,
Speaker 2: it uses strong AS two fifty six encryption with Microsoft
Speaker 2: Managed keys by default. For extra control, you can bring
Speaker 2: your own keys byok using Azure KEYVOULT that's Customer managed
Speaker 2: keys or CMK. In transit, all communication uses Transport Layer
Speaker 2: Security TLUS one point two or higher.
Speaker 1: Okay, very robust security layers. What about the responsible AI
Speaker 1: side ensuring the models themselves behave safely.
Speaker 2: Microsoft has a whole responsible AI framework specifically adapted for
Speaker 2: generitive models like those in Azure open Ai. It's generally
Speaker 2: a four stage approach. First, identify potential harms. This involves
Speaker 2: extensive testing, including red teaming, where experts actively try to
Speaker 2: make the model produce harmful output. Second, measure quantify how
Speaker 2: often and how severely these harms occur using metrics in
Speaker 2: human review. Third, mitigate implement tools and strategies to reduce
Speaker 2: those harms. This includes things like prompt engineering, guardrails, the
Speaker 2: content filters we discussed, designing user experiences carefully, maybe adding
Speaker 2: citations or limiting response line. It's a layered defense and.
Speaker 1: Depth strategy, defense and depth right.
Speaker 2: And fourth, operate. This is about having plans for ongoing
Speaker 2: monitoring after deployment, collecting telemetry, gathering user feedback, and having
Speaker 2: incident response plans ready if something goes wrong.
Speaker 1: So it's an ongoing process, not just a one time
Speaker 1: check exactly.
Speaker 2: Continuous vigilance for general operations. As your monitor is key.
Speaker 2: It collects activity logs, resource logs, performance metrics from your
Speaker 2: Azure open AI deployments. You can track things like processed
Speaker 2: inference tokens to see your usage or if you're using
Speaker 2: provision throughput metrics like Provision Managed Utilization V two show
Speaker 2: how efficiently you're using that reserve capacity. You query all
Speaker 2: this data using Cousto query language KQL okay.
Speaker 1: And finally, for scaling up, what about quotas and limits
Speaker 1: and this PTUM thing you mentioned right, So.
Speaker 2: The standard pays you go. As your open AI uses
Speaker 2: shared GPU infrastructure. Because it's shared, there are quotas and
Speaker 2: rate limits to ensure fair usage. There are limits on say,
Speaker 2: how many resources as you can deploy per region, how
Speaker 2: many concurrent DELI requests, how many whisper requests per minute,
Speaker 2: a number of fine tuning jobs, et cetera. Your main
Speaker 2: throughput limit is usually measured in tokens per minute or TPM.
Speaker 2: These TPM limits vary based on the region, the specific model,
Speaker 2: and your deployment type. Enterprise agreement customers often get higher
Speaker 2: default quotas requests per mint. RPM is directly related, typically
Speaker 2: six rpm per one thousand tpm.
Speaker 1: So you manage these TPM limits yes, As.
Speaker 2: Your open AI Quota management lets you allocate your total
Speaker 2: available quota across your different model deployments as needed. Now,
Speaker 2: for businesses needing really consistent high performance or low latency,
Speaker 2: especially for critical apps, that's where provision throughput Unit Managed
Speaker 2: or PQUM comes in.
Speaker 1: P TUM. What does that give you?
Speaker 2: It allows you to reserve dedicated processing capacity just for
Speaker 2: your models. This guarantees consistent performance, stable latency and throughput
Speaker 2: because you're not competing for resources on the shared infrastructure.
Speaker 2: It also often comes with a significant cost saving round
Speaker 2: fifty percent potentially compared to pay as you go for
Speaker 2: the same level as stained usage. You can buy PTUs
Speaker 2: hourly or commit to longer term reservations for.
Speaker 1: Even better rates and guarantees.
Speaker 2: Yes, importantly, PTUM comes with strong slas a ninety nine
Speaker 2: point nine percent uptime guarantee and a nine to nine
Speaker 2: percent token generation latency guarantee, which provides that predictability crucial
Speaker 2: for demanding applications.
Speaker 1: What an incredible journey we've taken today, Seriously, we went
Speaker 1: from the absolute fundamentals, what large language models even are,
Speaker 1: understanding tokens, embeddings, transformers. Then we dove straight into Azure
Speaker 1: open Ai, looking at how Microsoft packages these powerful models
Speaker 1: for enterprise use, covering security, compliance, all those critical aspects.
Speaker 1: We explored the really advanced stuff too, embeddings, vector databases,
Speaker 1: RGY for grounding models, multimodal capabilities with GPT four to
Speaker 1: oh function calling the assistance API. And we didn't forget
Speaker 1: the human element. Mastering communication through prompt engineering from basic
Speaker 1: principles to advanced techniques like chain of thought and react.
Speaker 1: Finally wrapping up with how to securely operationalize everything, compliance filtering, monitoring, quotas,
Speaker 1: and those PTUs for dedicated performance. It's a lot, but
Speaker 1: hopefully a clear picture emerged.
Speaker 2: It really covers the spectrum from the core concepts to
Speaker 2: the practicalities of building and running real world AI solutions
Speaker 2: responsibly on Azure. It's clear the power isn't just the
Speaker 2: tech itself, but how you wield it.
Speaker 1: That's perfectly put as you've hopefully gathered from our chat.
Speaker 1: The real magic happens when you understand not just what
Speaker 1: these AI tools can do, but how to guide them,
Speaker 1: how to integrate them thoughtfully, securely, and responsibly. So here's
Speaker 1: something to think about. Given how fast AI is moving
Speaker 1: and how white spread it's becoming, how might your understanding
Speaker 1: of these tools, the capabilities, the limitations, the ways to
Speaker 1: interact change how you tackle problems, not just tech problems,
Speaker 1: but any challenge where information, creativity and communication are key.
Speaker 2: Yeah, think about the possibilities that might have opened up
Speaker 2: just from this discussion. What new approaches could you take, What.
Speaker 1: New questions does the spark for you? Maybe you want
Speaker 1: to dig even deeper into the source material we use.
Speaker 1: Or perhaps you're already thinking what specific area should we
Speaker 1: deep dive into next time Food for thought. Until then,
Speaker 1: keep asking the big questions.