Problem Description

You can experience Event ID 4412 Storm from Source DFSR with the following Message : "The DFS Replication service detected that a file was changed on multiple servers. A conflict resolution algorithm was used to determine the winning file. The losing file was moved to the Conflict and Deleted folder." and your boot volume can run out of free space on Domain Controllers with DFS Replication installed.

Error Message

Log Name:      DFS Replication
Source:        DFSR
Date:          09/10/2021 5:55:18 PM
Event ID:      4412
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      DC.Domain.local
Description:
The DFS Replication service detected that a file was changed on multiple servers. A conflict resolution algorithm was used to determine the winning file. The losing file was moved to the Conflict and Deleted folder.

Below a screenshot of the related Event Storm :

Root Cause Analysis

A few days ago one of our customers asked for help because he run very fast out of free disk space on his Domain Controllers. I immediately requested from him to run the free TreeSize Tool to figure out what is filling up the disk so quickly.

Once we run the TreeSize Tool we saw that the \ConflictAndDeleted folder under DFSR is increasing 100MB every 1 Minute.

C:\System Volume Information\DFSR\Private\{UR934RU-JDJ984S-JDEW89D-8789RR}\ConflictAndDeleted

Below a screenshot of the free TreeSize Tool :

Our first action was to free up some space on the C:\ Volumes on all affected Domain Controllers so that they can continue with his normal operations. At the time of the problem we had nearly 8% to 22% free space on different affected Domain Controllers.

After some search on the Internet we saw the article mentioned below from Microsoft and decided to run the cleanup command on the affected Domain Controllers :

The ConflictAndDeleted folder size may exceed its configured limitation

wmic /namespace:\\root\microsoftdfs path dfsrreplicatedfolderinfo where "replicationgroupname=''" call cleanupconflictdirectory

Once done we then checked the DFS Replication Event Logs in Event Viewer and saw that 1 Second nearly up to 30 Event ID 4412 gets logged. Even if it is an Informational Event it is not normal to get so much Informational Events with the same Event ID every 1 Second.

Afterwards we checked the DFS Health Status and also tried to replicate some new files by placing the files into the Replication folder to figure out if it is working fine and saw that the replication works as expected on our Domain Controllers.

Furthermore we checked the files, which were reported as changed in Event ID 4412, but none of the files were changed or modified based on the "Date Modified" Date in Explorer. The files files were nearly 4-5 Month old.

We then decided to compare the HASH Value of the files between the Domain Controllers with Get-FileHash Powershell Command.

Get-FileHash -Path

BINGO !

All files reported in Event ID 4412 had a different HASH Value on the Domain Controllers. This means basically that a Remote or Local Scanning Software or Agent is continuously changing the HASH Value of the files. As a result DFS thinks that the file has been really changed and an endless replication begins between the Domain Controllers.

We then run Process Monitor Tool from Microsoft to figure out which Process is changing the HASH Value of the files in DFS Replication folder and saw that CyberArk Endpoint Privilege Manager Agent (vf_agent.exe) continuously doing changes to the DFSR folder.

Below a screenshot of the related ProcMon Logs :

Solution

We decided to remove CyberArk Endpoint Privilege Manager Agent (vf_agent.exe) from all of our Domain Controllers and BINGO !

After nearly 30-60 Minutes Event ID 4412 Storm stopped on all of our Domain Controllers. It can take some time till the winners file mentioned in Event ID 4412 gets replicated as a last time to all Domain Controllers and stops continuously replicating the same file again but it should not take more then 90 Minutes.

Important !

Please be aware that even if you add the DFSR Folder to the Exception List in CyberArk Endpoint Privilege Manager Agent it can still access, scan and do changes inside the DFSR Folder because CyberArk Agent has an Artificial Intelligence and can decide by itself if it sees an high activity or anomaly inside an folder.

Good luck !