No Response Ping

When running a discovery for network devices in SCOM 2012 R2 from a server running Server 2012 R2 Core, the discovery may fail with the above error message. This doesn’t seem to occur with a full installation of Server 2012 R2. The solution, as per this guide is to enable the pre-defined firewall rules. To do this in PowerShell:

Get-NetFirewallRule | ? { $_.DisplayName -match "operations" -and $_.Enabled -ne "True" } | Set-NetFirewallRule -Enabled "True"

I know that querying and setting the Enabled property in this way seems odd, but this property is an enum rather than a Boolean.

After setting the above rules, run the discovery again and the devices should be detected.

Matthew Long’s post on the useful DayTimeExpression module of the System.ExpressionFilter data filter is the only useful information I could find on its use, save for the MSDN documentation. However, in using the module, I’ve noticed what appears to be a bug in its implementation.

The problem

The module works by checking whether a date-time is bound (or not bound, as defined by <InRange />) by a window that’s defined by (a) a days of the week mask and (b) a time window, the start and end of which are specified by the number of seconds since midnight. The problem appears to be that the date-time is interpreted as local time, despite being defined in ISO 8601 format, e.g. 2015-05-29T17:15:00Z, rather than being correctly interpreted as UTC and converted to local time before being checked against the defined window. This problem occurs whether a date-time is manually specified, e.g. <Value Type="DateTime">2015-05-29T17:15:00Z</Value>, or is obtained at runtime, e.g. <XPathQuery Type="DateTime">./@time</XPathQuery>.

The workaround

It may be possible to solve this problem using XQuery/XPath, but my solution is to return the current local date-time from the PowerShell script called by the data source, albeit presented as UTC in ISO 8601 format. For example, if the script were executed at 17:15 (local time), the script would return, in its property bag, a property with a value of 2015-05-29T17:15:00Z. This is a hack, as, because I’m on BST (UTC+1), this value should be 2015-05-29T16:15:00Z, but this triggers the bug.

In my PowerShell script:

$Bag.AddValue("Date-time executed (local time)", (Get-Date).ToString("yyyy-MM-ddTHH:mm:ssZ"))

In my monitor type:

<Expression>
	<DayTimeExpression>
		<ValueExpression>
			<XPathQuery Type="DateTime">Property[@Name='Date-time executed (local time)']</XPathQuery>
		</ValueExpression>
		<StartTime>$Config/SecondsFromMidnightCheckWindowStart$</StartTime>
		<EndTime>$Config/SecondsFromMidnightCheckWindowEnd$</EndTime>
		<Days>$Config/DaysOfWeekMask$</Days>
		<InRange>true</InRange>
	</DayTimeExpression>
</Expression>

I am now able to define the window using the number of seconds from midnight, local time, and System.ExpressionFilter correctly checks this window against the execution time.

Daily state aggregations appear to be stored against the wrong date in the data warehouse database when the time zone of the server hosting the SQL Server instance is at an offset to UTC. This is evident in the UK, because for approximately half the year the UK is on UTC, at which point the daily aggregations are stored against the correct date, but during BST (UTC+1) they are stored against the previous date. I suspect this may only affect time zones at a positive offset to UTC, and that the problem is in dbo.StandardDatasetAggregate, possibly related to the calculation of @IntervalStartDateTime, but I haven’t been able to pinpoint the bug. I also haven’t checked whether the same situation exists with the other aggregate types, such as performance.

The problem can be best illustrated with a script, which works against a database running in the UK for web application transaction monitor-data generated in 2014:

USE OperationsManagerDW

-- Determine a web application transaction monitor with relevant data, i.e. during BST and where an unhealthy state has been recorded.
DECLARE	@ManagedEntityMonitorRowId INT,
		@Date DATE

SELECT TOP 1
	@Date = sdf.[Date],
	@ManagedEntityMonitorRowId = sdf.ManagedEntityMonitorRowId
FROM
			dbo.vStateDailyFull sdf

INNER JOIN	dbo.vManagedEntity me
ON			sdf.ManagedEntityRowId = me.ManagedEntityRowId
AND			me.FullName LIKE 'WebApplication[_]%'
AND			me.FullName NOT LIKE '%WatcherComputersGroup'

WHERE
	sdf.MonitorRowId =	(
							SELECT TOP 1
								MonitorRowId
							FROM
								dbo.vMonitor
							WHERE
								MonitorSystemName = 'System.Health.EntityState'
						)
AND	(
			sdf.InYellowStateMilliSeconds > 10000
		OR	sdf.InRedStateMilliseconds > 10000
	)
AND	sdf.[Date] BETWEEN '31/Mar/2014' AND '25/Oct/2014'		-- ~BST.
ORDER BY
	sdf.[Date]

-- Return the data.
SELECT
	*
FROM
	dbo.vStateFull
WHERE
	ManagedEntityMonitorRowId = @ManagedEntityMonitorRowId
AND	[DateTime] >= @Date
AND	[DateTime] < DATEADD(d, 2, @Date)
ORDER BY
	[DateTime]

For a particular web application transaction monitor, my results for the daily aggregation (AggregationTypeId = 30) for 31/03/2014 show that InYellowStateMilliseconds is 206173, but the sum total of InYellowStateMilliseconds for the hourly aggregations (AggregationTypeId = 20) is 0 for this date. The figure of 206173 shows against the hourly aggregations associated with the following day, 01/04/2014.

I have logged this bug on Microsoft Connect.

Edit: 13/08/2015: This bug has been fixed in management pack version 6.6.0.0.

---

I have discovered a bug in the latest SQL Server 2014 SCOM management pack (6.5.1.0).

The problem is that file groups are not discovered for databases containing filestreams or partition schemes, due to a bug in the discovery script. The health service subsequently goes on to discover the files associated with all the file groups, regardless of whether the file group has been discovered, and forwards the data to the management server. Upon receipt of the discovery data for the files, the management server rejects it because some of the files are associated with file groups that haven’t yet been discovered, i.e. the management server is unable to map some of the files to file groups. The result is that all files for the database aren’t discovered and are therefore not monitored.

The symptoms are event ID 10801 on management servers, when the management server processes the discovery data, and missing database file groups and files in the inventory. The cause is a bug in the DiscoverSQL2014FileGroups.js script in the Microsoft.SQLServer.2014.Discovery management pack, where it only accepts file group types FX and FG, but, according to MSDN, there are two more possible values: FD and PS.

I have logged this bug on Microsoft Connect. Please vote for this bug if it’s a problem for you.

I’ve implemented a workaround, until Microsoft resolve the problem, which restricts file and log file discovery to those file group types that have previously been discovered, i.e. FX and FG. This allows monitoring of these file group types, but does not monitor the missing types. Unfortunately I can’t distribute the updated management pack as it contains Microsoft code.

The cause of an error code of 2147954430 from a web application transaction monitor can be difficult to determine, especially as it’s often intermittent. In my case, investigating the watcher node revealed that too many long-running calls had been included in the monitor and this had caused a backlog on the node, as described by event ID 10503 in the Operations Manager log:

The HTTP URL Monitoring Module detected that a backlog of processing has happened. It might be an indication of too many URL monitors configured for this watcher node.

The solution was to increase the interval between executions of the monitor to one that more realistically reflected the likely execution time. And it might be useful to create a rule to raise alerts from these events on the watcher nodes.