Recently, the group led by Professor Nanlin Wang from the Beijing Academy of Quantum Information Sciences (BAQIS) and International Center for Quantum Materials, School of Physics, Peking University, in collaboration with the groups led by Professor Kaihui Liu and Professor Enge Wang from School of Physics, Peking University, has made important progress in the research of nonlinear optical properties of two-dimensional layered materials. They discovered huge higher-order nonlinear optical responses in a new type of van der Waals layered material - transition metal phosphorus chalcogenide MPX₃ (M = transition metal element, X = S, Se), which manifests great scientific and applicational value in the research of nonlinear optics. The work entitled "Giant nonlinear optical wave mixing in a van der Waals correlated insulator" were published online in Science Advances (Science Advances 10, adn6216 (2024)) on August 2, 2024.

Nonlinear optics is one of the most important fields in optics. Nonlinear optical wave mixing, such as high-order harmonics, sum and difference frequency generation, plays a crucial role in traditional fields including laser generation and manipulation, photon detection, and optical sensing, and also in emerging fields such as quantum photonics, quantum information, and on-chip nano-photonics. Currently, van der Waals layered two-dimensional materials attract intense research interest as new nonlinear optical materials. Compared with traditional three-dimensional bulk nonlinear optical materials, two-dimensional materials are friendly for nano-fabrication and -integration and have larger nonlinear optical coefficients, demonstrating great application prospects in state-of-the-art fields such as quantum photonics and quantum computing. Discovering efficient two-dimensional nonlinear optical materials and fabricating new types of nonlinear optical devices have become the frontiers of nonlinear optical research.

The research team found that transition metal phosphorus chalcogenide MPX₃ is a new type of two-dimensional nonlinear optical material hosting profound nonlinear optical responses and high nonlinear conversion efficiency, rendering its remarkable prospects in nonlinear optics in both scientific and applicational perspective. MPX₃ is a van der Waals antiferromagnetic insulator which recently received extensive attention from field of condensed matter physics. It hosts novel properties. Including rich correlations between charge, spin, and lattice subsystems, excitonic many-body quantum states, and tunable magnetic and optical properties. The complex correlated physical properties of the MPX₃ family distinguish it from the existing two-dimensional nonlinear optical materials which are simple physical systems. It provides intriguing possibility for studying the complex nonlinear light-matter interaction in complex systems, and achieving manipulation of optical functionality through multiple degrees of freedom.

Although there have been many studies on the optical properties of the MPX₃ family, its higher-order nonlinear optical properties have never been discovered. The researchers for the first time reported its profound higher-order optical nonlinearity. They irradiated MnPSe₃ and MnPS₃ samples with two near-infrared femtosecond lasers and found that the samples generated a series of higher-order nonlinear optical wave mixing signals (third harmonic generation, non-degenerate four-wave mixing, and six-wave mixing signals) spanning over ultraviolet to visible spectral range. The researchers measured the third-order nonlinear optical susceptibility for the third harmonic generation process in MnPSe₃. The near-infrared third-order susceptibility value reaches near the highest record values of 2D materials. Besides, the series of nonlinear wave mixing signals show extraordinary nonlinear conversion efficiencies, exceeding those of typical nonlinear optical materials (such as LiNbO₃, GaSe, WS₂). These results all illustrate the excellent nonlinear optical performance of this material system. The researchers further constructed a special nonlinear optical waveguide with multi-color, multi-directional emission, which may inspire future applications in photonic integration and quantum information processing.

The findings provide a new two-dimensional nonlinear optical material with correlated physical properties and enrich the understanding of the nonlinear optical properties of two-dimensional materials. The work brings intriguing opportunities for nonlinear optical applications and the development of new optical devices, and provides an excellent platform for exploring the nonlinear light-matter interaction in complex correlated material systems.

Figure 1 (a) Photograph of the nonlinear four-wave and six-wave optical mixing signals of the MnPSe₃ sample excited by two near-infrared femtosecond lasers, and schematic diagrams of the nonlinear optical conversion processes. (b) Spectra of different nonlinear wave mixing signals. (c) Third-order nonlinear optical susceptibility of MnPSe₃, compared to GaSe sample with typically large third-order susceptibility. (d) Schematic of the multi-color, multi-directional nonlinear optical waveguide.

The first authors of this paper are assistant researcher Li Yue from the BAQIS and associate researcher Chang Liu from School of Physics, Peking University. The corresponding authors are Li Yue, Tao Dong, associate researcher from School of Physics, Peking University, and Nanlin Wang, chair professor of PKU and a team leader of BAQIS. This research work was supported by the National Natural Science Foundation of China, the National Key Research and Development Program, the China Postdoctoral Science Foundation.

Link：https://www.science.org/doi/10.1126/sciadv.adn6216