XML LOG File Parsing

 Home  Previous  Next

Monitoring the content of an XML LOG file with Monitoring Studio

Specifying the file to monitor

1.Right-click the main Monitoring Studio icon or your previously-created application icon > KM Commands > New > File monitoring and analysis…
2.Specify that file is a LOG file, i.e. that Monitoring Studio needs to monitor only the new lines that are being added to the file.
3.Specify the path to the file. You can use wildcards (* and ?) if the name of the file changes over time (like a time-stamped LOG file). In such case, Monitoring Studio monitors the most recently updated file which matches with the specified path.
4.Enter a label and ID for the icon which is going to be created in the PATROL Console.

You have successfully setup the monitoring of an XML log file. Now to parse this file, you first need to preprocess the XML text in order to then run string searches or perform numeric value extraction on the result.

Pre-processing the content of the file (converting XML to CSV)

Right-click the File object which has been created > KM Commands > New > Text Pre-Processing… Select the Convert XML to CSV option.

EX_TextPreProcess_1Welcome

Selecting a Type of Conversion to Apply to a Log File

In this example, the records in this XML LOG file are in the following format:

<rec>

<vm>su37sr72</vm>

<ts>2003-09-22 11:47:35.511 CEST</ts>

<level>ERROR</level>

<class></class>

<method></method>

<ctx>

<pid>A141607</pid>

<appid>frontnet</appid>

<cname>User_3_0.getDefaultUserRole</cname>

<reqid>2</reqid>

<sesid>1uEPHTdRG2mM6GCfhv1EkwcBrCi68ffGizgIEtGHWFMt5Hc7lwE7!-1625978434!-1455528670!7501!7502!1064223951289</sesid>

<thrid>ExecuteThread: '68' for queue: 'default'-f7c8b25c01</thrid>

<cthid>ExecuteThread: '68' for queue: 'default'-f7c8b1696c</cthid>

</ctx>

<msg>

<![CDATA[FNNotAuthorizedException;FEA002002;No authorization to execute service operation]]>

</msg>

<exc>

<ts>2003-09-22 11:47:35.509 CEST</ts>

<sev>ERROR</sev>

<ctx>

<pid>A141607</pid>

<appid>frontnet</appid>

<cname>User_3_0.getDefaultUserRole</cname>

<reqid>2</reqid>

<sesid>1uEPHTdRG2mM6GCfhv1EkwcBrCi68ffGizgIEtGHWFMt5Hc7lwE7!-1625978434!-1455528670!7501!7502!1064223951289</sesid>

<thrid>ExecuteThread: '68' for queue: 'default'-f7c8b25c01</thrid>

<cthid>ExecuteThread: '68' for queue: 'default'-f7c8b1696c</cthid>

</ctx>

<stack>

<![CDATA[com.csg.pb.frontnet.exec_arch.calx.FNNotAuthorizedException: No authorization to execute service operation

at com.csg.pb.frontnet.services.user_3_0.bean.UserBean_3_0.getDefaultUserRole(UserBean_3_0.java:345)

at com.csg.pb.frontnet.services.user_3_0.bean.UserBean_3_0_3c05dc_EOImpl.getDefaultUserRole(UserBean_3_0_3c05dc_EOImpl.java:145)

at com.csg.pb.frontnet.services.user_3_0.bean.UserBean_3_0_3c05dc_EOImpl_WLSkel.invoke(Unknown Source)

at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:360)

at weblogic.rmi.cluster.ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:93)

at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:329)

at weblogic.rmi.internal.BasicServerRef.dispatch(BasicServerRef.java:178)

at weblogic.rmi.internal.ServerRequest.sendOneWayRaw(ServerRequest.java:92)

at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:112)

at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:263)

at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:230)

at weblogic.rmi.internal.ProxyStub.invoke(ProxyStub.java:35)

at $Proxy1401.getDefaultUserRole(Unknown Source)

at com.csg.pb.frontnet.services.user_3_0.bean.UserCA_3_0.getDefaultUserRole(UserCA_3_0.java:244)

at com.csg.pb.frontnet.apps.common.brokers.FnUserBroker.getDefaultPortalUserSettings(FnUserBroker.java:56)

at com.csg.pb.frontnet.apps.common.brokers.FnUserBroker.getDefaultMandant(FnUserBroker.java:280)

at com.csg.pb.frontnet.apps.common.base.DefaultAuthorizationStrategy.resetBusinessUnitParam(DefaultAuthorizationStrategy.java:72)

at com.csg.pb.frontnet.apps.common.base.DefaultAuthorizationStrategy.assertAvailableBusinessUnit(DefaultAuthorizationStrategy.java:48)

at com.csg.pb.frontnet.apps.common.base.AuthorizationServlet.doGetManaged(AuthorizationServlet.java:99)

at com.csg.cs.servlet.CSServlet.doGet(CSServlet.java:82)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)

at com.csg.cs.servlet.CSServlet.service(CSServlet.java:459)

at com.csg.pb.frontnet.util.servlet.FNServlet.service(FNServlet.java:334)

at com.csg.pb.frontnet.apps.common.base.CommonServlet.service(CommonServlet.java:66)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:262)

at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:21)

at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)

at com.csg.cs.security.wls.enforce.IntranetPLEnfFilter.doFilter(IntranetPLEnfFilter.java:174)

at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)

at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:287)

at com.csg.pb.frontnet.apps.common.util.RequestForwarder.forward(RequestForwarder.java:48)

at com.csg.pb.frontnet.apps.portals.base.PortalSelectionController.processSelectionPage(PortalSelectionController.java:49)

at com.csg.pb.frontnet.apps.portals.base.PortalSelectionController.doGetManaged(PortalSelectionController.java:29)

at com.csg.cs.servlet.CSServlet.doGet(CSServlet.java:82)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)

at com.csg.cs.servlet.CSServlet.service(CSServlet.java:459)

at com.csg.pb.frontnet.util.servlet.FNServlet.service(FNServlet.java:334)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:262)

at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:21)

at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)

at com.csg.cs.security.wls.enforce.IntranetPLEnfFilter.doFilter(IntranetPLEnfFilter.java:174)

at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)

at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:2684)

at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2412)

at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:140)

at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:121)

]]>

</stack>

</exc>

</rec>

So, <REC> is the XML tag for each new record. Let’s say that we would like to retrieve the <TS> value, the <LEVEL> value, the <CNAME> value under <CTX> and the <MSG> value.

Therefore, we are specifying that REC is the XML tag for a new record and that we would like to include the value for the following properties and sub-tags:TS LEVEL CTX.CNAME MSG. Please note the syntax "CTX.CNAME" which means the value of CNAME under the CTX tag.

EX_TextPreProcess_2Param

Defining the Conversion Parameters

Then, specify a label and ID for the text pre-processing object that will be created under the file icon.

EX_TextPreProcess_3ObjectDefinition

Defining the Object

As a result, we get a new icon in the PATROL Console, corresponding to the XML to CSV pre-processing:

EX_TextPreProcess_4TreeView

Accessing the Result File

This object has a single TransformResult text parameter as a result of the XML to CSV pre-processing:

2003-09-22 11:47:35.511 CEST;ERROR;User_3_0.getDefaultUserRole; FNNotAuthorizedException;FEA002002;No authorization to execute service operation]]>;

2003-09-22 11:52:05.984 CEST;ERROR;User_3_0.getDefaultUserRole; FNNotAuthorizedException;FEA002002;No authorization to execute service operation]]>;

2003-09-22 12:06:18.272 CEST;ERROR;User_3_0.getDefaultUserRole; FNNotAuthorizedException;FEA002002;No authorization to execute service operation]]>;

2003-09-22 12:09:53.920 CEST;ERROR;User_3_0.getDefaultUserRole; FNNotAuthorizedException;FEA002002;No authorization to execute service operation]]>;

2003-09-22 12:10:39.557 CEST;ERROR;KycBeneficialOwnerProfiles; FNDBDataAccessFailureException;RDS001003;Code not Found - TableName: Landcode_1_RefTableObject, BusinessUnit: 0012, Language: 891, Code: 001]]>;

2003-09-22 12:10:39.566 CEST;ERROR;KycBeneficialOwnerProfiles; FNDBDataAccessFailureException;RDS001003;Code not Found - TableName: Landcode_1_RefTableObject, BusinessUnit: 0012, Language: 891, Code: 001]]>;

2003-09-22 12:10:56.637 CEST;ERROR;CIFS_Customer_1.getCustomer; FNDBDataAccessFailureException;RDS001002;Code not Found - TableName: Service_Status_InfoRefTableObject, BusinessUnit: 0000, Code: CIFS_Customer_1_0]]>;

2003-09-22 12:10:56.643 CEST;SEVERE;CIFS_Customer_1.getCustomer; FNServiceNotAvailableException;FEA000001;Service not available - Service FNServiceState.getState]]>;

2003-09-22 12:10:56.945 CEST;ERROR;BPST_UserProfile_3.getUsers; FNDBDataAccessFailureException;RDS001002;Code not Found - TableName: Service_Status_InfoRefTableObject, BusinessUnit: 0000, Code: BPST_UserProfile_3_0]]>;

2003-09-22 12:10:56.950 CEST;SEVERE;BPST_UserProfile_3.getUsers; FNServiceNotAvailableException;FEA000001;Service not available - Service FNServiceState.getState]]>;

2003-09-22 12:21:30.004 CEST;ERROR;User_3_0.getDefaultUserRole; FNNotAuthorizedException;FEA002002;No authorization to execute service operation]]>;

Searching for strings in the result of the XML-to-CSV pre-processing

Right-click the Text-Pre-Processing icon > KM Commands > New > String search…

Now, we are going to search for lines that contain the string "SEVERE" in the second column, corresponding to XML records whose <LEVEL> is "SEVERE".

EX_TextPreProcess_5StringSearch

Performing a String Search Command on a Converted File

We enter SEVERE in the field for the string to search for, and select the option in the following column number and enter 2. Then we click the column separator tab and uncheck all except the semicolon and click Accept.
Finally, we click Next and follow the wizard steps as for any normal string search on a LOG file, keeping the default values shown.

This brings us to the last panel, where we select the I want to use the default thresholds option.

EX_TextPreProcess_6StringSearchConfirm

String Search Confirmation

As a result, we get the following String search object: SEVERE under the XML-to-CSV pre-processing object.

EX_TextPreProcess_7StringSearchTreeView

Accessing the String Search Results

NoteYou can create as many string searches as you want on a file and on a text-processing object, and you can create several different text-processing objects on the same file object.