Cache schema data

Some applications benefit from maintaining a persistent cache of schema definitions for a Dataverse organization. For example:

• Applications that need to work while disconnected from the Dataverse server.
• Applications that are sensitive to limited network bandwidth, such as mobile applications.

Dataverse has a lot of schema definition data, and few applications must track all of it. Limiting the amount of data to cache is essential.

The RetrieveMetadataChanges message provides two capabilities:

1. Query: Compose a single query to retrieve just the schema data you need. Composing queries is covered in Query Schema Definitions.
2. Cache Management: If you cache the schema definition data with your app, you can use RetrieveMetadataChanges to efficiently retrieve only the changes since your last query. Use the information about these changes to add or remove items in your cache. Caching schema data might result in a significant improvement time for your application to start, and is the focus of this article.

After you define a query for the schema data you want to cache, use other capabilities of the RetrieveMetadataChanges message to create and maintain a cache of Dataverse schema data.

Create a cache

As described in Query Schema Definitions, the first step is to create a query that defines the kind of schema information you're interested in. Then use the RetrieveMetadataChanges message to execute that query and cache the schema definitions returned in your application. Find a way to persist this data that is appropriate for the needs of your app. You might save it as a file that your application reads when it starts. You also need to save data about the last time you ran the query. Structure your cache in a way that makes sense for your application. To manage removal of deleted items, it's important that your cache uses the MetadataId as a unique identifier. For more information, see Remove deleted items.

Detect changes

No event exists to detect when schema definition changes occur. Refresh the cache based on the needs of your application. Usually, you retrieve changes when the application starts. But you might poll the system regularly if you want to detect changes over time. Regardless of your strategy, keep track of the time you sent the previous request.

The RetrieveMetadataChangesResponse.ServerVersionStamp property contains information about the point in time the request to RetrieveMetadataChanges occurred. Take the ServerVersionStamp value from the previous response and use it as the value for the RetrieveMetadataChangesRequest.ClientVersionStamp when you send it again by using the same query.

When you include the ClientVersionStamp property in the request, the RetrieveMetadataChangesResponse.EntityMetadata property returned contains only the changed or added schema data since the previous request. You can add them to your cache.

If you also include the DeletedMetadataFilters parameter, any schema items deleted since the previous request are included in the RetrieveMetadataChangesResponse.DeletedMetadata property. You can remove them from your cache.

This result is smaller and faster than executing the original query again.

Manage expired cache

Dataverse stores information about changes for 90 days by default. This value is stored in the Organization.ExpireSubscriptionsInDays property. If you send a request with a ClientVersionStamp value that's older than the setting value, Dataverse returns an ExpiredVersionStamp error (0x80044352). Be prepared to handle this specific error and reinitialize your cache when it occurs.

Add changed items

Just as when you perform your initial query to initialize your cache, the RetrieveMetadataChangesResponse.EntityMetadata property returned for subsequent requests by using RetrieveMetadataChangesRequest.ClientVersionStamp includes the full hierarchy of changed items.

Each item in the hierarchy has a nullable boolean HasChanged value. When this value is false, it means the current item didn't change, but something below it in the hierarchy changed. When the HasChanged value is true, it means the current item in the hierarchy changed.

Tracking options

One area that tracking changes might not seem intuitive is how options for choice columns are tracked. This area is a good example to understand the HasChanged property.

When you add a new option to a choice column, the following data in the hierarchy is returned through the RetrieveMetadataChangesResponse.EntityMetadata property:

Remove deleted items

When you include both ClientVersionStamp and DeletedMetadataFilters parameters for a RetrieveMetadataChanges message, the RetrieveMetadataChangesResponse.DeletedMetadata property contains data about any deleted items. This property is a DeletedMetadataCollection that contains a set of Keys and Values. The keys are DeletedMetadataFilters enum values and you can use these values to access a subset of the deleted values.

The deleted values are returned as a collection of Guid values. The values for deleted items aren't filtered based on your query. Many Guid values aren't in your cache, but the Guid values for your cached item are included. You must look for any matching Guid values and remove those items. Disregard any values that aren't in your cache.

Tracking deleted options

Notice that the DeletedMetadataFilters enum contains a member for OptionSet, but not option. If you remove any options from a choice column, you won't find a reference to a specific option that was deleted. You need to go to your cached options and compare them to the current options returned for that OptionSet.

See also

Web API Query schema definitions and detect changes Sample (C#)
SDK for .NET Query schema definitions and detect changes Sample (C#)
Query Schema Definitions