Because the speed of the Infinity Fabric is tied to the integrated memory controller (IMC), we now have the answer to why using higher frequency memory has a disproportionally large big impact on performance. Due to the way the Zen architecture is designed, memory speed has a much larger impact on performance than what you see with Intel's monolithic processors. At the same time, it could also be part of the reason why Ryzen has issues with high-frequency and low-latency DDR4.
According to AMD, it is a 256-bit wide bi-directional crossbar. Think of it as town-square for the chip, where tagged data and instructions change hands between the various components. Within the CCX, the L3 cache performs some inter-core connectivity. The speed of the Infinity Fabric crossbar on a "Summit Ridge" Ryzen processor is determined by the memory clock. When paired with DDR4-2133 memory, for example, the crossbar ticks at 1066 MHz (SDR, actual clock). Using faster memory, according to AMD, hence has a direct impact on the bandwidth of this interconnect.
AMD is working on improving DDR4 memory support and is expected to roll out a microcode update in May. Motherboard makers need to implement this update via motherboard firmware updates.