Modius Data Center Blog

I am routinely surprised at how difficult it can be to determine the total energy consumption for many data centers. Stand-alone data centers can at least look at the monthly bill from the utility, but as the Green Grid points out when discussing PUE metrics, continuous monitoring is preferred whenever possible. Measurement in an environment where resources, such as chilled water, are shared with non-data center facilities can be even more complex. I’ll discuss that topic in the coming weeks. For now, I want to look just at the stand-alone data center.

In general, the choices are pretty simple for a green-field installation. The only real requirement is commitment to buying the instrumentation. Solid-core CTs are cheaper, and generally smaller for the same current range. Wiring in the voltage is easy. Retrofits are more interesting. Nobody likes to work on a hot electrical system, but shutting down a main power feed is a risky process, even with redundant systems.

One logical metering point is the output of the main transfer switches. Many folk assume they already have power metering on their ATS. It has an LCD panel showing various electrical readings, after all. Unfortunately, more often than not, only voltage is measured. That’s all the switch needs to do its job. Seems that the advanced metering option is either overlooked or the first thing to go when trimming the budget.

Retrofitting the advanced option into an ATS is not trivial. Clamping on a few CTs might not seem tough, but the metering module itself generally has to be completely swapped out. Full shut-down time.

A separate revenue-grade power meter is not terribly expensive these days. In some cases it may even be competitive with the advanced metering option from your ATS manufacturer. Meters that include power-quality metrics such as THD can be found for less than $3K, CTs included. Such a meter could be installed directly on the output of the ATS, but the input of the main distribution panel is generally a better option.

Clamping on the CTs is relatively straightforward, even on a live system, though it can be tricky if the cabling is wired too tightly. Slim, flexible Rogowski coils are an excellent option in this case. A bit pricier, but ease of installation can make back the difference in labor pretty quickly.

For voltage sensing, distribution panels often have spare output terminals available. This is ideal in a retrofit situation, and desirable even in a new install. Odds are the breaker rating is higher than the meter could handle, so don’t forget to include protection fusing. If no spare circuit is available, you can perhaps find one that is at least non-critical, such as a lighting circuit, and could be shut down long enough to tie in the voltage.

Worst-case retrofit scenario, you have no local voltage connections available. CTs alone are better than nothing. A good monitoring system can combine those readings with nominal voltages, or voltages from the ATS, to provide at least apparent power. Most meters can be powered from a single-phase voltage supply, even 110V wall power. I recommend springing for the full power meter even in this case. At some point you’ll likely have some down time, hopefully scheduled, on this circuit, and you can perform the full proper wiring at that time.

The final decision about your meter is whether to get the display. If your goal is continuous measurement (i.e., monitoring), the meter should be communicating with a monitoring system. The LED or LCD display will at best provide you a secondary check on the readings. The option also complicates the installation, because you need some kind of panel mounting to hold it and make it visible. It can become more of a space issue than one might expect for a 25-sq. inch display. Avoiding the full display output saves on the cost of the meter, and saves even more on the installation labor.

Look for a meter with simple LEDs or some other indicator to help identify wiring problems like mis-matched current and voltage phases. If the meter is a transducer only, have the monitoring system up and running, and communication wiring run, before installing the meter, so you can use its readings to troubleshoot the wiring. Nobody wants to open that panel twice!

Continuous monitoring of total power is critical to managing the capacity and efficiency of a data center. Whether your concern is PUE, carbon footprint, or simply reducing the energy bill, the monthly report from the utility won’t always provide the information you need to identify specific opportunities for improvement. Even smart meters might not be granular enough to identify short-term surges, and won’t allow you to correlate the data with that from other equipment in your facility. It’s hard to justify skimping on one or two meters for a new data center. Even in a retrofit situation, consider a dedicated meter as an early step in your efficiency efforts.

You may have asked yourself, “Why do I need another monitoring and reporting product if I already have five?”  True, you most likely don’t need another monitoring product, but rather what you really, really need is a system to link these systems together. 

Why?  Because several different monitoring systems operating in their own silos doesn’t help you improve your business.  Instead, what you need to do is build business logic for optimization and capacity expansion strategies, as well as decrease the time spent to repair problems. 

To do this effectively, you need a super system: what we call the “mother of all monitors”.  This is a system that cannot only collect a superset of monitoring data from different point solutions, but also connect directly to other devices that may not currently be monitored (e.g. generators, transfer switches, breaker panels, etc.).  And it needs to do this with the kind of scalability, analytics, and ability to integrate with other management systems that you would expect from an enterprise-class tool. 

Here at Modius, we are already seeing this happen in the field.  There is a current trend among data center managers to link their monitoring platforms together so that they have one common central platform to view and navigate to distributed monitoring systems.  We have designed our application, OpenData, with a  “Monitor of Monitors” architecture in order to provide operators with a single pain of glass into both the facilities infrastructure including power-chain, cooling, and redundancies as well as IT system level information.

The key problems solved are:

1.  System-level metrics - Link system level IT metrics to facilities capacities  
2.  Trouble shooting - Accelerate trouble shooting and fault dependency mapping
3. Alarm management - Reduction in “noise-level” alarms
4. Analytics - Building business-level metrics (BI) for capacity, efficiency, etc.
5. Controls-based integrations – Improved automation based on broad data capture

Here is some more detail on each of these benefit areas …

1)      System-level metrics

Typically, IT system-level metrics are collected by system management tools and will provide logical properties based on MIB-2 or the Host MIB (RFC-1514).  This provides IT managers with data on the operating health of the equipment and capacity related to CPU, Disc, I/O, Memory.  What management systems typically do not provide, however, is how facilities (power, cooling, etc.) impacts the cost of operations and the amount of optimal cooling. 

By linking IT system-level metrics with unified facilities monitoring through a single portal, higher level business and operating metrics can be formulated to reduce the cost of operations by tuning available cooling resources to the actual needs of each server instance or other IT gear.

2)      Trouble shooting

By consolidating event and performance data into a single view, you can quickly determine the cascade of failures with the visibility to determine the impacts of facility equipment.  An example could be a PDU failure and what devices are in the path of the affected circuit.  In redundant environments there will be a fail-over to the second PDU but in most cases the assurances of a successful hand-off are difficult to predict.  By linking both facilities BMS, PDU’s, UPS, Genset with system level IT information the relationships are documented, visualized, correlated and actively monitored.

3)      Reduction in rogue alarms

By linking point solutions and consolidated even level data, a complete historical view may be achieved.  Through this historical view, alarm flows can be optimized and reduced operationally.  An example would be a BMS received alarms at a rate where the alarms become noise as they are not easily tuned.  Also contextually, it is very difficult to look at what a typical operating condition is as there is not enough or broad enough history to proactively set truly meaningful thresholds or deviations.

4)      BI-based business metrics

With a single point of consolidation, you can quickly build reports and dashboards across platforms.  An example would be a stock chart type view when you can visualize a period of time.  This is used to determine deviations from the norm which might cause downtime or affect operational performance.  With several independent systems it becomes impossible to correlate based on time or carry enough history to gain the insight necessary to prevent a potential outage.

5)      Single application launch point

The “Monitor of Monitor” architecture brings a unified structure to gain access to operational and control systems.  An example use case would be to identify cooling requirements based on broad-based data capture (e.g. an array of environmental sensors at the rack level, or real-time server-inlet temperatures taken directly from servers themselves) and then tie the resulting performance metrics into building control systems to tune VFD’s and cooling output.  Integrating the BMS application directly to the monitoring system allows the use the real-time data required and feedback mechanism to optimize cooling and cost without overheating the IT equipment.

Conclusion

If you would like more detail on how Modius can help with any the above topic areas, please reach out directly using info@modius.com, and we will be happy to set up an appointment.

Here at Modius, we are seeing an increasing number of requests among Co-locations (Co-los) and Managed Service Providers (MSPs) to help them capture more robust and accurate power measurement data.  In one sense, this trend is nothing new because all data centers—whether captive inside an enterprise or an outsourced service provider—need accurate power measurement, typically for improving:

• Capacity optimization
• Energy efficiency
• Uptime assurance

But we find that Co-lo’s and MSP’s have a special need that takes power reporting to the next level: Providing disaggregated energy consumption and power usage data by customer at a very granular level, often by rack or even a group of servers.  Typically, they need detailed power metering for each customer, principally for:

• More accurate customer billing
• Detailed status reporting to the customer (in real-time) through a customer portal

Customers are now wanting this information not only to be sure their power bills are accurate, but also to try and determine their available power capacity, usage trends, and accurate data to support reporting on PUE and Carbon management.  Or even more of a challenge, they need to unify data across different locations because their customers are spread across several different buildings. 

Theoretically, some of this data can been captured from the servers.  In fact, with distributed systems management tools, reporting on server energy consumption (at the server level) is relatively commonplace.  But this data source is incomplete.  What if you want to factor in cooling and other related energy consumption?  Or what if you also want environmental reporting for bottom/middle/top for each rack?  Now, this is much more challenging …

In general, most Co-Lo’s don’t have access to the server instrumentation data at the chassis level.  And in terms of power and cooling, we’ve found that most co-location providers are still struggling to unify a broad range of equipment into a single monitoring fabric and extend the framework across disparate systems and locations. 

Happily, there are several Co-Lo’s operators taking the initiative by unifying their monitoring of power and cooling equipment with a real-time data center monitoring and measurement system like Modius OpenData.  And many are augmenting power and cooling data by installing new breaker level metering and.  Moreover, many are even using this data to create centralized customer portals to provide their customers with reporting and a real-time view of their power capacity and consumption.  Further, they are adding a layer of analytics and baselines on energy efficiency and reliability. 

As the industry becomes more competitive, service providers cannot continue with business as usual.  Many Co-lo’s and MSP’s have taken this initiative so that they can differentiate themselves, have better visibility on how they can extend their internal resources, and provide PUE and Carbon reporting services to their customers. 

We believe the underlying driver behind this trend is the fact that an increasing number of corporations and enterprises with large IT departments are being tasked by their senior management to provide comprehensive reports on power usage and their relative efficiency, regardless of whether the enterprise owns their own data center facilities or outsource part of their infrastructure. 

Be it end-users, Co-lo’s or MSP’s, everyone is increasingly looking to software providers like Modius to solve the comprehensive measurement and reporting problem, and we believe they are finding that Modius OpenData is the right product at the right time and value.