When you are working with the Identity Sync service in K2, you should understand the components and functions of each component.
||Responsible for querying the Identity Providers (IdP) that store the identity information (for example AD, AAD, and SharePoint) and sending the results to the Sync Engine.
||Executes the Providers, and then transforms and stores the results in an intermediary table (intermediate cache).
||Transforms the Sync Engine data to identities (for example users, groups, group membership and managers) that K2 can consume, and stores them in the Identity schema.
||The Identity Schema is the runtime cache of identities that are consumed by K2.
Full Sync vs Differential Sync
A full sync (or the initial Sync), syncs all IdP data at that moment in time. The system saves a change token (also known as a delta link/watermark) to keep track of this state. If the Sync is successful the system saves the watermark, ensuring that no IdP data is incorrect because of failures.
Subsequent syncs will be differential syncs, sending the change token to the IdP to request only the changes made after the previous Sync. Differential syncs are fast and efficient if they occur frequently, and K2 recommends that the sync schedule should be as short as possible (the default is 15 minutes).
Only AD, AAD and SharePoint support differential syncs. Other providers such as LDAP, custom User Managers and SQLUM do a full sync every time. If you have many identities in SQLUM for example, each scheduled sync takes as long to run as the initial sync. Take that into consideration when configuring schedules for IdPs that do not support differential syncs. You may want to schedule syncs less frequently for IdPs that do not have volatile user data.
The SmartObjects used by the Identity Sync Service are available in K2 Management under System > Sync Service.
||Use the Run History SmartObject method to see the status and time that your last sync ran.
||Use this SmartObject to manually start a full sync, to set an ongoing sync schedule or to get a list of existing schedules. When performing the initial sync, setting the Skip Deleted property to yes will not sync users deleted from the IdP. Performing the initial sync and setting the Skip Deleted property to no will sync deleted users, but in a disabled state. If these users' accounts change, the changes will sync when the sync schedule triggers.
||Use the methods of the Provider Instance SmartObject to list existing provider instances and see associated information about them. Execute the List Provider Instances method to see provider instance names and additional information. Use the provider instance name to start a full sync and to scheduled ongoing sync.
||Use the methods of the Provider SmartObject to list existing providers and to see associated information. Execute the List Providers method to see provider names, types, and if they are enabled.
K2 recommends doing the following SQL maintenance before and after the initial Sync:
- Database backup and shrink
- Check indexes and re-index/reorganize index.
Database backup and shrink
- Right-click K2 database and select Tasks > Back Up
- Ensure that the Backup type is on Full and click OK to complete the backup.
- Right-click K2 database and select Tasks > Shrink > Database
- Ensure that it’s the correct database and click OK to complete the shrink.
Check indexes and re-index / reorganize index
Although K2 does have the built-in capability to re-index data, we suggest doing the re-indexing/reorganizing of indexes one-by-one, especially on PROD environments, to minimize the performance impact on the environment.
SQL query to check index fragmentation:
SELECT dbschemas.[name] AS [Schema]
,dbtables.[name] AS [Table]
,dbindexes.[name] AS [Index]
,[ActionRequired] = CASE
WHEN indexstats.avg_fragmentation_in_percent > 5
AND indexstats.avg_fragmentation_in_percent < 30
THEN 'Reorganize Index'
WHEN avg_fragmentation_in_percent > 30
THEN 'Rebuild Index'
FROM sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL, NULL) AS indexstats
INNER JOIN sys.tables dbtables ON dbtables.[object_id] = indexstats.[object_id]
INNER JOIN sys.schemas dbschemas ON dbtables.[schema_id] = dbschemas.[schema_id]
INNER JOIN sys.indexes AS dbindexes ON dbindexes.[object_id] = indexstats.[object_id]
AND indexstats.index_id = dbindexes.index_id
WHERE indexstats.database_id = DB_ID()
AND dbschemas.[name] = 'SyncEngine'
ORDER BY indexstats.avg_fragmentation_in_percent DESC
For the above query, for every ActionRequired that states Reorganize Index, you need to run the following query:
ALTER INDEX [<Index name>] ON [SyncEngine].[<Table name>] REORGANIZE;
For the above query, for every ActionRequired that states Rebuild Index you need to run the following query:
DBCC DBREINDEX ('SyncEngine.<Table name>', '<Index name>');
When troubleshooting issues with the Identity Sync service, try to identity where the error is occurring in the stack of components. The table below lists some common issues or items to check on each component in the stack.
- Is there a change token error/unauthorised error from SharePoint, AAD, AD, etc.?
- The issue is in the Provider if the Sync schema and the IdP data does not match up.
- For on-prem: check the SyncEngine.RunHistory table for Sync failures (or use the Management SMO - Run History)
- For Cloud Ops: check Badger.SyncAudit table for any failures
- Is Sync not completing?
- Is it a more generic error while syncing – Sync engine bug
- For on-prem: check SyncEngine.RunHistory for Sync failures (Management SMO)
- For cloud Ops: check Badger.SyncAudit table for failures
- Check HostServer Logs for errors
- Check ETW logs for errors
- Run the Script to verify that ETL has completed (SyncEngineGetETLStatus)
- Check ETW logs for errors
|When something is wrong, but there are no errors
- Exactly what happened before running a Sync?
- Was the opt-in installer run? Was it run before/after?
- Are the flags correct (ResolvingEnabled = true/false)?
- Compare data from IdP to SyncEngine to Identity.Identity, are there differences?
When logging a ticket:
- Run through the troubleshooting steps above and provide the info you discovered.
- Provide the HostServer.UpdateHistory data
- If there are deadlocks, provide deadlock graphs (in SQL profiler)
- If known, provide the Environment flavor and setup (multi-node/single domain/on-prem using AAD/etc.)?
- What are the Provider types or combinations of provider types you are using?
- Screenshots from the IdP vs screenshots of Sync schema/Identity schema
- SyncEngine schema and Identity schema (obfuscation script if required)
- Script to check for stale data:
- Script to check status of ETL