Introduction

In this tutorial I will show you how to collect and analyze a SCOM ETL Trace in depth. Collecting an ETL Trace is pretty easy and a straight forward action.

The most important part comes afterwards.

Once the ETL Trace is collected which of the tenth of ETL Traces should we analyze and what to look for ?

In this tutorial I will mainly focus on this question and show you how I analyze the ETL Traces and what kind of tools I use.

Overview at a glance

1. How to collect a SCOM ETL Trace on the SCOM Management Server ?
2. How to collect a SCOM ETL Trace on the SCOM Agent ?
3. Which Tools are used to analyze a SCOM ETL Trace ?
4. Which ETL Trace files should I analyze ?
5. What to look for inside an ETL Trace ?

1. How to collect a SCOM ETL Trace on the SCOM Management Server ?

1.1. Open a cmd window as Administrator on the Server where you open the Operations Manager Console. Afterwards please close the Operations Manager Console.

1.2. Navigate to the "\Tools" folder.

SCOM 2012 R2 = C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Tools
SCOM 2016 = C:\Program Files\Microsoft System Center 2016\Operations Manager\Server\Tools
SCOM 2019 = C:\Program Files\Microsoft System Center\Operations Manager\Server\Tools

1.3. Type "StopTracing" without quotation marks and press enter.

1.4. Now type "StartTracing VER" without quotation marks and press enter. Please be aware that the parameter VER is in CAPITAL letters.

1.5. Restart the “System Center Management Configuration Service” on the server and open the Operations Manager Console new.

1.6. Now reproduce the problem.

1.7. After the reproduction, please run the command "StopTracing" in cmd window.

1.8. After this command please run the command "FormatTracing" and wait until it finishes. This can take some time.

1.9. Once the ETL trace has been formatted into a human readable format a explorer window to the "\OpsMgrTrace" folder will automatically open. The exact path to the "\OpsMgrTrace" folder is mentioned below.

1.10. Please save the content of the folder.

Additional Info :

Trace file Path in SCOM 2007 = C:\WINDOWS\Temp\OpsMgrTrace
Trace file Path in SCOM 2012, 2016 and 2019 = C:\Windows\Logs\OpsMgrTrace

2. How to collect a SCOM ETL Trace on the SCOM Agent ?

StartTracing, StopTracing and FormatTracing are CMD files. The mentioned above CMD files in “STEP 1” are also available on the SCOM Agent .

You basically need to navigate to the Agent Trace Tool Path "C:\Program Files\Microsoft Monitoring Agent\Agent\Tools" and apply the same actions on the Agent in “STEP 1” above, if you need an Agent Trace too.

If the problem you are experiencing is related to an Agent, in example if you do an action on the SCOM Management Server which should affect an Agent and if it does not, than it would be good idea to collect an SCOM Agent Trace too.

Basically a Trace from both sides would be required.

3. Which Tools are used to analyze a SCOM ETL Trace ?

SCOM ETL Trace are mostly very big in size therefore if you reproduce a problem to collect an Trace you should be very fast. I mostly place the RDP windows from the Agent and SCOM Management Server side by side to not waste time during the reproduction of the problem.

Basically if you collect a trace you should be very fast otherwise you can get easily a 1GB ETL Trace.

There are several Trace Analysis Tools available on the market also inside the Support Tools which you can use.

Most of them are crashing if you open too big files. Therefore I am using NotePad++ because this tool can open 1GB big text files very easily.

If you need to search for some specific keywords or errors inside all trace files then I would strongly recommend you to use the free tool from MythicSoft named “Agent Ransack“. This tool can index very fast all trace files and quickly find what you are looking for.

4. Which ETL Trace files should I analyze ?

Once the ETL Trace has been formatted, the "\OpsMgrTrace" folder looks like below. As you can see below there are 3 file types available inside the folder.

*.etl (Native ETL Trace files.)
*.log (Human readable, formatted Trace files.)
*.sum (A summary and the amount of the Health Service Event Reference occurrences inside the *.log files.)

Important Notice !

Once you saved and copied the whole "\OpsMgrTrace" folder you can proceed as follows. All file names which begins with the "Previous.*" name can be deleted no matter if the file type extension is "*.ETL" or "*.LOG".

These files belongs to the timeline before you stopped the SCOM ETL Trace in “STEP 1” with the "StopTracing" command. SCOM continuously collects a light weight ETL Trace for troubleshooting purposes but these Trace files are mostly not enough to figure out the root cause of the problem.

All "*.ETL" files can be deleted too because SCOM doesn’t delete the native Trace files once you format it. There is no need for these saved files anymore.

Based on experience and as an Engineer who analyzed maybe 1000 times or more a SCOM ETL Trace, I can say that in 99% of the cases the "TracingGuidsNative.log" file is mostly enough to figure out the root cause of the problem.

The "TracingGuidsNative.log" file includes 99% of the information which are available inside the other log files too. This is the main file where you should take a look first.

Directory of C:\Windows\Logs\OpsMgrTrace
05/01/2019 08:19 PM.
05/01/2019 08:19 PM..
05/01/2019 08:08 PM 131,072 Previous.TracingGuidsAdvisor.etl
05/01/2019 08:09 PM 242 Previous.TracingGuidsAdvisor.log
05/01/2019 08:09 PM 716 Previous.TracingGuidsAdvisor.log.sum
05/01/2019 08:08 PM 22,085,632 Previous.TracingGuidsAPM.etl
05/01/2019 08:09 PM 1,644,898 Previous.TracingGuidsAPM.log
05/01/2019 08:09 PM 1,249 Previous.TracingGuidsAPM.log.sum
05/01/2019 08:08 PM 65,536 Previous.TracingGuidsApmConnector.etl
05/01/2019 08:10 PM 0 Previous.TracingGuidsApmConnector.log
05/01/2019 08:10 PM 603 Previous.TracingGuidsApmConnector.log.sum
05/01/2019 08:08 PM 65,536 Previous.TracingGuidsBID.etl
05/01/2019 08:10 PM 0 Previous.TracingGuidsBID.log
05/01/2019 08:10 PM 594 Previous.TracingGuidsBID.log.sum
05/01/2019 08:08 PM 65,536 Previous.TracingGuidsConfigService.etl
05/01/2019 08:10 PM 0 Previous.TracingGuidsConfigService.log
05/01/2019 08:10 PM 604 Previous.TracingGuidsConfigService.log.sum
05/01/2019 08:08 PM 65,536 Previous.TracingGuidsDAS.etl
05/01/2019 08:11 PM 0 Previous.TracingGuidsDAS.log
05/01/2019 08:11 PM 594 Previous.TracingGuidsDAS.log.sum
05/01/2019 08:08 PM 2,293,760 Previous.TracingGuidsFailover.etl
05/01/2019 08:11 PM 36,044 Previous.TracingGuidsFailover.log
05/01/2019 08:11 PM 1,916 Previous.TracingGuidsFailover.log.sum
05/01/2019 08:08 PM 720,896 Previous.TracingGuidsManaged.etl
05/01/2019 08:12 PM 27,168 Previous.TracingGuidsManaged.log
05/01/2019 08:12 PM 703 Previous.TracingGuidsManaged.log.sum
05/01/2019 08:08 PM 65,536 Previous.TracingGuidsNASM.etl
05/01/2019 08:12 PM 0 Previous.TracingGuidsNASM.log
05/01/2019 08:12 PM 597 Previous.TracingGuidsNASM.log.sum
05/01/2019 08:08 PM 44,105,728 Previous.TracingGuidsNative.etl
05/01/2019 08:12 PM 531,508 Previous.TracingGuidsNative.log
05/01/2019 08:12 PM 3,356 Previous.TracingGuidsNative.log.sum
05/01/2019 08:08 PM 65,536 Previous.TracingGuidsOMEDService.etl
05/01/2019 08:13 PM 0 Previous.TracingGuidsOMEDService.log
05/01/2019 08:13 PM 602 Previous.TracingGuidsOMEDService.log.sum
05/01/2019 08:08 PM 65,536 Previous.TracingGuidsScript.etl
05/01/2019 08:13 PM 0 Previous.TracingGuidsScript.log
05/01/2019 08:13 PM 597 Previous.TracingGuidsScript.log.sum
05/01/2019 08:08 PM 393,216 Previous.TracingGuidsUI.etl
05/01/2019 08:14 PM 2,734 Previous.TracingGuidsUI.log
05/01/2019 08:14 PM 1,343 Previous.TracingGuidsUI.log.sum
05/01/2019 08:08 PM 131,072 TracingGuidsAdvisor.etl
05/01/2019 08:14 PM 212 TracingGuidsAdvisor.log
05/01/2019 08:14 PM 704 TracingGuidsAdvisor.log.sum
05/01/2019 08:08 PM 458,752 TracingGuidsAPM.etl
05/01/2019 08:15 PM 10,890 TracingGuidsAPM.log
05/01/2019 08:15 PM 2,363 TracingGuidsAPM.log.sum
05/01/2019 08:08 PM 65,536 TracingGuidsApmConnector.etl
05/01/2019 08:15 PM 0 TracingGuidsApmConnector.log
05/01/2019 08:15 PM 592 TracingGuidsApmConnector.log.sum
05/01/2019 08:08 PM 851,968 TracingGuidsBID.etl
05/01/2019 08:15 PM 44,512 TracingGuidsBID.log
05/01/2019 08:15 PM 984 TracingGuidsBID.log.sum
05/01/2019 08:08 PM 1,048,576 TracingGuidsConfigService.etl
05/01/2019 08:16 PM 218,440 TracingGuidsConfigService.log
05/01/2019 08:16 PM 7,099 TracingGuidsConfigService.log.sum
05/01/2019 08:08 PM 65,536 TracingGuidsDAS.etl
05/01/2019 08:16 PM 0 TracingGuidsDAS.log
05/01/2019 08:16 PM 583 TracingGuidsDAS.log.sum
05/01/2019 08:08 PM 65,536 TracingGuidsFailover.etl
05/01/2019 08:17 PM 0 TracingGuidsFailover.log
05/01/2019 08:17 PM 588 TracingGuidsFailover.log.sum
05/01/2019 08:08 PM 65,536 TracingGuidsManaged.etl
05/01/2019 08:17 PM 0 TracingGuidsManaged.log
05/01/2019 08:17 PM 589 TracingGuidsManaged.log.sum
05/01/2019 08:08 PM 65,536 TracingGuidsNASM.etl
05/01/2019 08:17 PM 0 TracingGuidsNASM.log
05/01/2019 08:18 PM 586 TracingGuidsNASM.log.sum
05/01/2019 08:08 PM 1,245,184 TracingGuidsNative.etl
05/01/2019 08:18 PM 843,024 TracingGuidsNative.log
05/01/2019 08:18 PM 8,302 TracingGuidsNative.log.sum
05/01/2019 08:08 PM 65,536 TracingGuidsOMEDService.etl
05/01/2019 08:18 PM 0 TracingGuidsOMEDService.log
05/01/2019 08:18 PM 591 TracingGuidsOMEDService.log.sum
05/01/2019 08:08 PM 65,536 TracingGuidsScript.etl
05/01/2019 08:19 PM 0 TracingGuidsScript.log
05/01/2019 08:19 PM 586 TracingGuidsScript.log.sum
05/01/2019 08:08 PM 65,536 TracingGuidsUI.etl
05/01/2019 08:19 PM 0 TracingGuidsUI.log
05/01/2019 08:19 PM 582 TracingGuidsUI.log.sum
78 File(s) 77,846,191 bytes
2 Dir(s) 76,715,843,584 bytes free

5. What to look for inside an ETL Trace ?

There isn’t a standard rule available but I mostly begin searching for the name of the component which experience the problem in the Operations Manager Console.

Imagine you have a problem with creating or saving a Subscription. Then its a good idea to begin with the name of the Subscription instead of looking for the keyword "Error". If you experience or see a pop up error message  in SCOM Console, seraching for this exact error message would be a good approach too.

The keywords "Error" or "Exception" are the last words in my search list.

Once you search for these keywords you will see that hundreds of errors and exceptions occurs in the background therefore beginning with some specific keywords will be a good approach.

Once you find the rows with the Subscription names you can take a look at the 100 rows above and below of the Subscription names for the keywords "Error" and "Exception" to figure out what happened before and what was the result afterwards.

I hope, that this gave you a clue what to look for inside a SCOM ETL Trace.

Good luck !