Roundtable: Liquid cooling in data centres

Roundtable sponsor

Cooling technology is a critical focus for data centres as their energy needs grow – and a shift to liquid cooling has brought considerations to the fore that didn’t exist a few years ago.

With rack densities continuing to increase, the industry is being asked to design for currently unknown future heat and load densities.

This shift to liquid cooling was explored at a roundtable held recently in central London by CIBSE Journal, supported by Tate. This article, the first of two on the event, looks at the challenges that this rapid transition poses for building services engineers. 

Liquid cooling is a high-efficiency thermal management strategy for data centres, leveraging the fact that water’s volumetric heat capacity is more than 3,400 times greater than air’s. As AI and high-density computing push rack power beyond the limits of air cooling, three primary methods emerge: direct-to-chip (cold plates on processors); immersion cooling (submerging servers in dielectric fluid); and rear-door heat exchangers.

These systems use a secondary loop (technology cooling system) to transfer heat from IT components to a primary facility water loop via a coolant distribution unit (CDU). This specific approach reduces energy consumption, shrinks the physical footprint and enables waste-heat reuse.

(l-r): Seddik Daoud, Aidan Jones and Felix Cox

The key reason for the growth in data centre’s energy needs is the introduction of Nvidia’s new Blackwell chips, which use up to 130kW of electrical power within a single server rack. They supersede the less powerful Nvidia Hopper chips, which consume around 20kW per rack.

It was possible to use air cooling with Hopper (100) chips, said James Maher, director of engineering at roundtable sponsors Tate, who told his fellow delegates that the advent of the powerful Blackwell (GB200/300) chips was a ‘game-changer’ that had kick-started the shift to liquid cooling.

While air cooling hasn’t disappeared, pointed out Sophia Flucker, technical director at MICIM Group, it now makes up a smaller proportion of data centres’ requirements because it cannot cope with the heat transfer required for these much higher-density racks.

Chris Dunn, principal mechanical engineer at Arup, noted how surprisingly quickly data centres had transitioned to liquid cooling. ‘The density has far surpassed what anybody could have predicted two or three years ago by an order of megawatts,’ he said.

Tate’s head of R&D, Aidan Jones, agreed. ‘The technology was there,’ he said. ‘We knew there was going to be that step change and it was coming, but we didn’t know it was going to get so high so quickly.’

Gavin Murphy and Barbara Smitten

Indeed, liquid cooling was adopted at such scale and speed that the sector was caught out, observed Barbara Smitten, critical systems associate director at Cundall. ‘It accelerated at such a pace that the industry had insufficient time to adapt and optimise design approaches,’ she said. ‘As these changes evolve, there is the need for standardisation and further optimisation.’

However, some clients are looking at a mix of air and liquid cooling systems, said Gavin Murphy, operations director of Ethos.

Sustainability was cited as a key factor in switching to liquid cooling. ‘The cooler you run the chips, the longer they can last for,’ said Felix Cox, associate director, data centres, at Aecom. ‘We can maybe run them warmer and get a bit of an energy saving, but then you’re potentially throwing away chips quicker and it’s hard to get hold of the minerals in those chips.’

Some participants recalled that liquid – that is, water – used to be de rigueur for cooling in data centres.

Matt Butler, associate at Arup, recalled that the first data centre he worked at, nearly 25 years ago, used chilled-water cooling.

From left: Gavin Murphy, Barbara Smitten, James Maher, Alex Smith and Matt Butler

As semi-conductor technology developed, however, chips became more efficient and water cooling largely disappeared from all but high-power compute, added Dunn: ‘The efficiency of chips got so much better that they didn’t need liquid.’

This bred an industry-wide fear about the presence of liquid near IT equipment, said Butler: ‘Everybody got scared.’

Seddik Daoud, associate director at consultancy FoundDigital, told the roundtable that higher-density chips are driving more compute into a given area of the data hall, known in the industry as ‘white space’. This is becoming a smaller proportion of the overall facility, as the supporting mechanical and electrical plant areas – known as grey space – grows to handle much higher power densities and liquid cooling.

‘We have small data halls compared with  the larger mechanical, electrical and public health plant areas,’ added Daoud. He pointed out that there is only so much cooling that can be put on a roof, for example, and these shifts in space requirements ‘fundamentally change’ what must be included when submitting schemes for planning permission.

‘That’s one of the biggest difficulties at the early stage,’ Daoud said. ‘Clients are asking you to guess where things are going and to build in flexibility for the unknown, which is not cheap.

‘Planning has become an expensive process: you cannot afford to get it wrong.’ 

‘The white space increasingly presents one of the biggest scheduling risks, given the number of disciplines working in close proximity,’ says Maher. ‘At Tate, we’re delivering infrastructure as pre-integrated systems, assembled and tested off site as a single solution, delivered modular, reducing onsite complexity, improving safety and creating a more predictable path
to handover.’

Simon Steed and Tim Hale

Simon Steed, director of design of the data centre division of the PM Group, agreed that liquid cooling meant a new approach, telling the roundtable: ‘The increasing interest has led to design revisions on several data centre projects that have already been able to secure planning and permit approval.’ 

Some major ‘hyper scale’ data centre projects have even attempted to change from air to liquid cooling without having to revise their planning application, which can take up to two years, said Aecom’s Cox .

However, installation of liquid cooling poses its own set of challenges, added Maher, such as the ‘intricate’ pipe work in confined mechanical corridors because of the presence of CDUs.

Structural point loads can also present problems when retrofitting existing buildings for liquid cooling, said Dunn, who added: ‘A modern high-capacity rack row requires so much mechanical and electrical infrastructure that many existing floors simply cannot support the weight.’

The sheer scale of electrical and mechanical services, including the fibre needed to run them, leads to ‘crazy’ structural point loads, Dunn told the roundtable. However, Daoud suggested a solution when retrofitting data centres with higher-density racks, where only half of the space is often used. ‘On refurbishment projects, the point loads are so high they’re putting spreader plates everywhere for the servers,’ he said. ‘That’s a workaround to deal with heavy point loads.’

The space freed up in data centres can lead, perhaps inevitably, to pressure to squeeze in more white space, Daoud added: ‘You can retrofit for liquid cooling, but you still have to make sure the electrical infrastructure can take the increased density.’ 

Simon Steed and Christopher Leahy

Chris Leahy, sector director, data centres, at Deerns, added that ‘retrofitting buildings for data centres can be somewhat challenging, particularly given the limitations with existing brownfield spaces, such as in cities’.

The discussion also highlighted another challenge – high-density chip setups create huge heat loads, which can have catastrophic consequences if the cooling system fails.

Rapid expansion and contraction of air can cause a room to implode, warned Butler.

‘If you get a break in cooling, you get massive swings in pressure in the room,’ he said. ‘As the air heats up, if you’re not careful, you get such low densities that you can implode a room.’

These kinds of effects, however, are peculiar to data centres, he added: ‘If I’m designing an office building, it [implosion] is never going to be a consideration.’

Chris Hayward and Sophia Flucker

Another cooling challenge when designing data centres is the 27°C temperature limit for data halls set out in ASHRAE guidance. The roundtable heard that this benchmark wasn’t driven by the technical limitations of servers or IT, but was established as an upper limit for human workers to safely operate in the space without violating labour directives.

Flucker said: ‘The 27°C is more to do with the server acoustic tests, which happen at 25°C… the server fans run at lower speed at this temperature; 27°C gives a bit of a buffer.’

Daoud added: ‘Especially in Europe, occupational health and safety rules limit how long staff can work in environments above 40°C, typically to around 30 minutes. A lot of regulations get triggered by hot environments.’

Like a lot of developers, data centre owners and operators face the Catch-22 issue that they cannot get Grid power without planning permission, but are unable to secure permission without a network connection. This is a problem across Europe, said PM Group’s Steed, noting ‘unrealistic’ timelines of up to nine years for some grid connections in Germany.

Sustainability of data centres is a growing issue for designers in Europe and further afield. The challenges of energy and water will be discussed in part two of the roundtable in next month’s CIBSE Journal. Participants will give their views on the future of data centres, including whether they have prospects in space.