Edit

Share via

Install execution providers with Windows ML

With Windows ML, certain execution providers (EPs) are dynamically downloaded, installed, and shared system-wide via the Windows ML ExecutionProviderCatalog APIs, and are automatically updated. To see what EPs are available, see Supported execution providers.

This page covers how to install EPs onto a user's device. Once installed, you'll need to register execution providers with ONNX Runtime before using them.

Install all compatible EPs

For initial development, it can be nice to simply call EnsureAndRegisterCertifiedAsync(), which will download and install all EPs available to your user's device, and then registers all EPs with the ONNX Runtime in one single call. Note that on first run, this method can take multiple seconds or even minutes depending on your network speed and EPs that need to be downloaded.

// Get the default ExecutionProviderCatalog
var catalog = ExecutionProviderCatalog.GetDefault();

// Ensure execution providers compatible with device are present (downloads if necessary)
// and then registers all present execution providers with ONNX Runtime
await catalog.EnsureAndRegisterCertifiedAsync();

Find all compatible EPs

You can see which EPs (including non-installed EPs) are available to the user's device by calling the FindAllProviders() method.

ExecutionProviderCatalog catalog = ExecutionProviderCatalog.GetDefault();

// Find all available EPs (including non-installed EPs)
ExecutionProvider[] providers = catalog.FindAllProviders();

foreach (var provider in providers)
{
    Console.WriteLine($"{provider.Name}: {provider.ReadyState}");
}

The execution providers returned will vary based on the user's device and available execution providers. On a compatible Qualcomm device without any execution providers currently installed, the above code outputs the following...

QNNExecutionProvider: NotPresent

Each ExecutionProvider has a ReadyState property that indicates its current status on the device. Understanding these states helps you determine what actions your app needs to take.

ReadyState Definition Next steps
NotPresent The EP is not installed on the client device. Call EnsureReadyAsync() to download and install the EP and add it to your app's runtime dependency graph.
NotReady The EP is installed on the client device, but hasn't been added to the app's runtime dependency graph. Call EnsureReadyAsync() to add the EP to your app's runtime dependency graph.
Ready The EP is installed on the client device and has been added to your app's runtime dependency graph. Call TryRegister() to register the EP with ONNX Runtime.

Install a specific EP

If there is a specific ExecutionProvider your app wants to use, and its ReadyState is NotPresent, you can download and install it by calling EnsureReadyAsync().

// Download and install a NotPresent EP
var result = await provider.EnsureReadyAsync();

// Check that the download and install was successful
bool installed = result.Status == ExecutionProviderReadyResultState.Success;

Installing with progress

The APIs for downloading and installing EPs include callbacks that provide progress updates, so that you can display progress indicators to keep your users informed.

Screenshot of a progress bar reflecting download progress

// Start the download and install of a NotPresent EP
var operation = provider.EnsureReadyAsync();

// Listen to progress callback
operation.Progress = (asyncInfo, progressInfo) =>
{
    // Dispatch to UI thread (varies based on UI platform)
    _dispatcherQueue.TryEnqueue(() =>
    {
        // progressInfo is out of 100, convert to 0-1 range
        double normalizedProgress = progressInfo / 100.0;

        // Display the progress to the user
        Progress = normalizedProgress;
    };
};

// Await for the download and install to complete
var result = await operation;

// Check that the download and install was successful
bool installed = result.Status == ExecutionProviderReadyResultState.Success;

Next steps

Now that you've installed execution providers, see Register execution providers to learn how to register them for usage with ONNX Runtime.

Production app example

For production applications, here's an example of what your app might want to do to give yourself and your users control over when downloads occur. You can check if new execution providers are available and conditionally download them before registering:

using Microsoft.Windows.AI.MachineLearning;

var catalog = ExecutionProviderCatalog.GetDefault();

// Filter to the EPs our app supports/uses
var providers = catalog.FindAllProviders().Where(p =>
    p.Name == "MIGraphXExecutionProvider" ||
    p.Name == "VitisAIExecutionProvider" ||
    p.Name == "OpenVINOExecutionProvider" ||
    p.Name == "QNNExecutionProvider" ||
    p.Name == "NvTensorRtRtxExecutionProvider"
);

if (providers.Any(p => p.ReadyState == ExecutionProviderReadyState.NotPresent))
{
    // Show UI to user asking if they want to download new execution providers
    bool userWantsToDownload = await ShowDownloadDialogAsync();

    if (userWantsToDownload)
    {
        // Download all EPs
        foreach (var p in providers)
        {
            if (p.ReadyState == ExecutionProviderReadyState.NotPresent)
            {
                // Ignore result handling here; production code could inspect status
                await p.EnsureReadyAsync();
            }
        }

        // And register all EPs
        await catalog.RegisterCertifiedAsync();
    }
    else
    {
        // Register only already-present EPs
        await catalog.RegisterCertifiedAsync();
    }
}

See also

  • Last updated on