Azure OpenAI, or the same models in a different wrapper

Azure OpenAI, or the same models in a different wrapper

Azure offers OpenAI's embedding models within Microsoft's cloud. The models are exactly the same, so result quality and dimension counts do not differ from direct access, while everything around them looks different.

That difference is why companies choose this route. Billing runs through an existing Microsoft agreement, data is processed in a region you choose, access is controlled by the same mechanism as your other resources, and traffic can travel over a private network without reaching the public internet.

This text covers what is specific to this access route. A description of the models themselves, choosing between them, and truncating dimensions sits in a separate piece.

A deployment instead of a model

This is the most common cause of errors when porting code and the thing that confuses practically everyone on a first attempt.

With direct access you supply a model name. Here you create a deployment, give it a name of your own, and use that name in calls. The deployment name can be anything and need have nothing in common with the underlying model's name.

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://resource-name.openai.azure.com",
    api_key=os.environ["AZURE_OPENAI_KEY"],
    api_version="2024-10-21",
)

response = client.embeddings.create(
    model="embeddings-production",
    input=chunks,
)

The value passed as the model is a deployment name rather than a model name. A deployment named this way may run a smaller or larger model underneath, and the code does not reveal which.

That carries an advantage and a drawback. The advantage is swapping models without changing code: you reconfigure the deployment and the application uses the new one. The drawback is that the code does not state what actually computes the vectors, so the model name and version must be recorded alongside the index, or six months later nobody will reconstruct it.

A practical convention: name deployments after the model and their purpose, combining the model name with the environment for instance. Names like "deployment-1" look innocent and tell nobody anything a year later.

Quotas, the thing to plan for

The second surprise concerns throughput, since it works differently from direct access.

Quota is assigned to a subscription separately for each region, each model, and each deployment kind, expressed in tokens per minute. When creating a deployment you allocate part of that pool to it, and the remaining availability drops accordingly.

A request rate limit follows automatically from the token quota, in a proportion set separately for each model. That means a deployment with a small token allocation accepts few requests per minute, even when each is short.

The practical consequence concerns indexing. Processing a large archive requires sending fragments in batches, and the request rate limit often binds before the token limit does. A sensible arrangement is batches of a few dozen fragments, several threads in parallel, and handling rate limit responses with exponential backoff.

Separate the deployments too. A dedicated deployment for indexing and another for serving queries means overnight archive processing does not take throughput from users asking questions during the day.

At larger scale a reserved throughput option exists, billed hourly rather than per token. It pays off only under steady high utilisation, so price it against your own traffic rather than assuming a reservation is always cheaper.

Regions and compliance

This is usually the real reason for choosing this route, so knowing exactly what you get pays off.

You create the resource in a specific region and processing happens there. Under personal data residency requirements that is sometimes a condition of entry that direct access does not satisfy.

A caveat deserves stating: not every model is available in every region. Check that before choosing a region, since moving a resource later means creating a new one and recomputing the index.

Access is controlled by the permission mechanism shared across the cloud, so a key in code can be replaced by an identity assigned to the service. That solves the problem of keys circulating through repositories and configuration files, and it is worth using rather than copying the pattern from direct access.

Traffic can be confined to a private network, through an endpoint reachable only from your virtual network. Requests then never reach the public internet at any point, which some security policies require outright.

With that configuration, think through the development environment straight away. A resource reachable only from the virtual network will not answer from a developer's machine, so local work needs either a connection into that network or a separate publicly reachable resource meant purely for development. The second is more convenient and requires watching that production data never lands in it.

Billing differences

The token rate here depends on the deployment kind, and that is the first thing to check. A global deployment, meaning one that does not constrain where processing happens, costs exactly what direct access costs. A deployment confined to a data zone runs about ten percent higher, and one pinned to a single region from ten to over thirty percent higher depending on the region chosen. That matters, because constraining where processing happens is the usual reason for coming here. A second difference gets skipped during planning.

Batch mode, halving the price in exchange for results delivered with a delay, is available with direct access and not on this route. That means a one off indexing of a large archive costs at least twice as much here, and more on a deployment pinned to a region.

The practical conclusion is sometimes surprising and deserves honest consideration. On an archive measured in billions of tokens the difference on the larger embedding model runs into hundreds of dollars, and into thousands at tens of billions, so one off indexing can be done more cheaply through direct access while leaving ongoing query serving here, if the compliance requirement concerns production traffic rather than archive processing itself.

That arrangement obviously needs checking with whoever owns compliance, since in some organisations the requirement covers all data regardless of when it is processed. It is worth knowing the option exists, though, rather than assuming the price is identical.

Add the surrounding costs to the bill too: the vector database, network traffic between services, and any private network endpoints, which carry their own hourly charge.

That last item tends to surprise, because it accrues regardless of traffic. A private network endpoint costs the same at one query a day as at a million, so on a small project it can exceed the cost of the embeddings themselves. With three environments, each holding its own endpoint, that item triples, even though two of them sit idle most of the day.

Porting an existing application

If you have working code using direct access, the move takes an afternoon provided you know four differences.

The first is the address. Instead of the vendor's shared endpoint you supply your own resource address, unique to your organisation and region.

The second is the interface version, stated explicitly when creating the client. That carries an advantage: a new version does not change your application's behaviour until you raise it yourself. It carries a drawback too: getting stuck on a version from two years ago is easy, since nothing reminds you.

The third is a deployment name instead of a model name, described above and responsible for most first run errors.

The fourth is authentication. A key works and is simplest, while the proper cloud answer is a service assigned identity, which removes storing a secret anywhere.

Take the opportunity to check whether your client library supports this route directly. Official libraries have a separate client class and switching amounts to swapping it. Intermediary libraries are sometimes less polished here, and some conflate the model name with the deployment name, producing errors hard to diagnose from your own code.

Model versioning and retirements

This part is specific to this access route and on a longer lived project matters more than everything above.

A deployment points at a specific model version, and versions are retired on an announced schedule. You receive notice in advance, while you must act, since a deployment on a retired version stops working.

An automatic update option to the newest version exists, and that choice deserves making deliberately. Automatic updating protects against a sudden outage and introduces a different risk: vectors computed with a new model version may not be fully comparable with the old ones, and the change happens without your involvement.

A sensible arrangement runs like this. Keep the deployment serving user queries on a pinned version and update it deliberately, alongside recomputing the index or after confirming the difference is negligible. Set a test deployment to update automatically, so you see in advance what a new version brings.

Record the date and model version at every index recomputation too. That is one line in the project documentation which, when somebody asks why search works worse than in March, saves a day of investigation.

Azure against the alternatives

The first row wins when an organisation already works in this cloud and has region or network requirements. Adding models to an existing agreement is then noticeably simpler than taking a new vendor through procurement and legal.

The third row deserves honest consideration, since it gets skipped. If the requirement reads "data does not leave our infrastructure", a model run in house satisfies it in a way no cloud can, whatever the contracts say.

Common mistakes

The first is passing a model name instead of a deployment name. Code ported directly from direct access returns an error whose text does not name the cause.

The second is one deployment for everything. Overnight archive indexing then takes throughput from users asking questions during the day.

The third is assuming a model is available in your chosen region. Not every model is available everywhere, and moving a resource later means recomputing the index.

The fourth is planning around batch mode. It is unavailable on this route, so one off indexing costs at least twice what the direct access price list suggests.

The fifth is keeping a key in code when a service assigned identity is available. The cloud's permission mechanism solves that better than any key rotation.

The sixth is not recording which deployment and which model produced the vectors. A deployment name does not state that by itself, and six months later nobody will reconstruct it.

FAQ

How does it differ from direct access?

The models are the same, so results do not differ. What differs is everything around them: the choice of processing region, access control through the cloud's permission mechanism, the option of confining traffic to a private network, and billing within an existing vendor agreement. The rate differs too: a global deployment costs what direct access costs, while one confined to a data zone or a single region costs noticeably more.

Why does code from direct access not work?

Most often because the call must supply a deployment name rather than a model name. You create the deployment yourself and give it any name, so the same application switched to this route needs that value changed along with the resource address and interface version.

Is batch mode available?

No, and that is a meaningful difference when planning costs. The half price discount for delayed results covers direct access, so a one off indexing of a large archive comes out at least twice as expensive here.

How should I plan quotas for indexing?

Quota is assigned in tokens per minute separately for region, model, and deployment kind, and a request rate limit follows from it in a model dependent proportion. On a large archive, create a separate deployment for indexing, send fragments in batches, and handle rate limit responses with exponential backoff.

Where should computed vectors live?

In a vector database, checking that it also meets your region requirements. A search service exists in the same cloud, and for smaller collections the options covered in the pieces on pgvector or Qdrant run in your own infrastructure will do.

Documentation sits on Microsoft's site, and quota management on a separate page.