Publications

Preprints

• Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents
  Zergham Ahmed, Kazuki Irie, Joshua B. Tenenbaum, Christopher J. Bates, Samuel J. Gershman
• Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View
  Gyuryang Heo, Timothy Ngotiaoco, Kazuki Irie, Samuel J. Gershman, Bernardo Sabatini

2026

• Sequential-Parallel Duality in Prefix Scannable Models
  Morris Yau, Sharut Gupta, Valerie Engelmayer, Kazuki Irie, Stefanie Jegelka, Jacob Andreas
  International Conference on Learning Representations (ICLR), April 2026

2025

• Fast weight programming and linear transformers: from machine learning to neurobiology
  Kazuki Irie, Samuel J. Gershman
  Transactions on Machine Learning Research (TMLR), December 2025
• Entorhinal cortex signals dimensions of past experience that can be generalised in a novel environment
  Sam Hall-McMaster, Lennart Wittkuhn, Luianta Verra, Noa L. Hedrich, Kazuki Irie, Peter Dayan, Samuel J. Gershman, Nicolas W. Schuck
  Journal of Neuroscience, December 2025 (in press)
• Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers
  Kazuki Irie, Morris Yau, Samuel J. Gershman
  Conference on Neural Information Processing Systems (NeurIPS), December 2025
• Overcoming classic challenges for artificial neural networks by providing incentives and practice
  Kazuki Irie, Brenden M. Lake
  Nature Machine Intelligence, October 2025
• Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph
  Kazuki Irie
  Findings of the Annual Meeting of the Association for Computational Linguistics (ACL Findings), short paper, July 2025
• Key-value memory in the brain
  Samuel J. Gershman, Ila Fiete, Kazuki Irie
  Neuron, March 2025
• Metalearning Continual Learning Algorithms
  Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
  Transactions on Machine Learning Research (TMLR), February 2025
  Early version appeared online in 2023 as “Automating Continual Learning.”

2024

• MoEUT: Mixture-of-Experts Universal Transformers
  Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber, Christopher Potts, Christopher D. Manning
  Conference on Neural Information Processing Systems (NeurIPS), December 2024
• Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers
  Lorenzo Tiberi, Francesca Mignacco, Kazuki Irie, Haim Sompolinsky
  Conference on Neural Information Processing Systems (NeurIPS), December 2024
• SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
  Róbert Csordás, Piotr Piękos, Kazuki Irie, Jürgen Schmidhuber
  Conference on Neural Information Processing Systems (NeurIPS), December 2024
• Self-Organising Neural Discrete Representation Learning à la Kohonen
  Kazuki Irie*, Róbert Csordás*, Jürgen Schmidhuber
  International Conference on Artificial Neural Networks (ICANN), September 2024
• Exploring the Promise and Limits of Real-Time Recurrent Learning
  Kazuki Irie, Anand Gopalakrishnan, Jürgen Schmidhuber
  International Conference on Learning Representations (ICLR), May 2024

2023

• Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions
  Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
  Conference on Empirical Methods in Natural Language Processing (EMNLP), short paper, December 2023
• Approximating Two-Layer Feedforward Networks for Efficient Transformers
  Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
  Findings of EMNLP (EMNLP Findings), December 2023
• Contrastive Training of Complex-Valued Autoencoders for Object Discovery
  Aleksandar Stanić*, Anand Gopalakrishnan*, Kazuki Irie, Jürgen Schmidhuber
  Conference on Neural Information Processing Systems (NeurIPS), December 2023
• Mindstorms in Natural Language-Based Societies of Mind
  Mingchen Zhuge*, Haozhe Liu*, Francesco Faccio*, Dylan R. Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, Louis Kirsch, Bing Li, Guohao Li, Shuming Liu, Jinjie Mai, Piotr Piękos, Aditya Ramesh, Imanol Schlag, Weimin Shi, Aleksandar Stanić, Wenyi Wang, Yuhui Wang, Mengmeng Xu, Deng-Ping Fan, Bernard Ghanem, Jürgen Schmidhuber
  NeurIPS Workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models (R0-FoMo), December 2023
  Best Paper Award @ R0-FoMo Workshop.
• Topological Neural Discrete Representation Learning à la Kohonen
  Kazuki Irie*, Róbert Csordás*, Jürgen Schmidhuber
  ICML Workshop on Sampling and Optimization in Discrete Space, July 2023
  Later version published at ICANN 2024.
• Accelerating Neural Self-Improvement via Bootstrapping
  Kazuki Irie, Jürgen Schmidhuber
  ICLR Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo), May 2023
• Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules
  Kazuki Irie, Jürgen Schmidhuber
  International Conference on Learning Representations (ICLR), May 2023

2022

• CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representations
  Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
  Conference on Empirical Methods in Natural Language Processing (EMNLP), short paper, December 2022
• Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules
  Kazuki Irie, Francesco Faccio, Jürgen Schmidhuber
  Conference on Neural Information Processing Systems (NeurIPS), November 2022
• Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks
  Kazuki Irie, Jürgen Schmidhuber
  NeurIPS Workshop on Memory in Artificial and Real Intelligence (MemARI), November 2022
• The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention
  Kazuki Irie*, Róbert Csordás*, Jürgen Schmidhuber
  International Conference on Machine Learning (ICML), July 2022
• A Modern Self-Referential Weight Matrix That Learns to Modify Itself
  Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber
  International Conference on Machine Learning (ICML), July 2022
• The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
  Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
  International Conference on Learning Representations (ICLR), April 2022
• Unsupervised Learning of Temporal Abstractions using Slot-based Transformers
  Anand Gopalakrishnan, Kazuki Irie, Jürgen Schmidhuber, Sjoerd van Steenkiste
  Neural Computation, March 2023

2021

• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
  Kazuki Irie*, Imanol Schlag*, Róbert Csordás, Jürgen Schmidhuber
  Conference on Neural Information Processing Systems (NeurIPS), December 2021
• The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
  Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
  Conference on Empirical Methods in Natural Language Processing (EMNLP), November 2021
• Linear Transformers Are Secretly Fast Weight Programmers
  Imanol Schlag*, Kazuki Irie*, Jürgen Schmidhuber
  International Conference on Machine Learning (ICML), July 2021
• A Modern Self-Referential Weight Matrix That Learns to Modify Itself
  Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber
  NeurIPS Workshop on Deep Reinforcement Learning (DeepRL), December 2021
  Later version published at ICML 2022.
• Improving Baselines in the Wild
  Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber
  NeurIPS Workshop on Distribution Shifts (DistShift), December 2021
• Learning Adaptive Control Flow in Transformers for Improved Systematic Generalization
  Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
  NeurIPS Workshop on Advances in Programming Languages and Neurosymbolic Systems (AIPLANS), December 2021
  Later version published at ICLR 2022.
• Unsupervised Learning of Temporal Abstractions using Slot-based Transformers
  Anand Gopalakrishnan, Kazuki Irie, Jürgen Schmidhuber, Sjoerd van Steenkiste
  NeurIPS Workshop on Deep RL (DeepRL) & Workshop on Offline RL (OfflineRL), December 2021
  Later version published in Neural Computation.
• Training and Generating Neural Networks in Compressed Weight Space
  Kazuki Irie, Jürgen Schmidhuber
  ICLR Workshop on Neural Compression, May 2021

Before 2020 — Language Modeling & Speech Recognition

Note: In my prior life as a Master’s and Ph.D. student (2013-2020), I intensively worked on language modeling before the current era of large language models or LLMs. My Ph.D. work has addressed several topics that are still popular in the field, including the development of large/deep (>100-layer) transformer language models (Interspeech 2019) concurrently to OpenAI’s GPT-2 in 2019 (note that we already had a 96-layer model—like GPT-3 released a year later in 2020), mixture of experts to scale up language models at Google/YouTube (ICASSP 2018), identifying and addressing the “KV-cache” memory problem of transformers (ICASSP 2020), distillation of large transformer language models into compact recurrent neural network language models (another ICASSP 2020; building off my earlier work on distilling LSTM language models into n-gram language models; ICASSP 2018), byte-level processing for modeling of low-resource languages with special sets of alphabets (ICASSP 2017), etc…

• Advancing Neural Language Modeling in Automatic Speech Recognition
  K. Irie
  PhD Thesis, Computer Science Department, RWTH Aachen University, Germany, May 2020
• The RWTH ASR system for TED-LIUM release 2: Improving Hybrid HMM with SpecAugment
  W. Zhou, W. Michel, K. Irie, M. Kitza, R. Schlüter, H. Ney
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), May 2020
• Domain Robust, Fast, and Compact Neural Language Models
  A. Gerstenberger, K. Irie, P. Golik, E. Beck, H. Ney
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), May 2020
• How Much Self-Attention Do We Need? Trading Attention for Feed-Forward Layers
  K. Irie, A. Gerstenberger, R. Schlüter, H. Ney
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), May 2020
  slides
• Training Language Models for Long-Span Cross-Sentence Evaluation
  K. Irie, A. Zeyer, R. Schlüter, H. Ney
  IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), December 2019
  poster
• A comparison of Transformer and LSTM encoder decoder models for ASR
  A. Zeyer, P. Bahar, K. Irie, R. Schlüter, H. Ney
  IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), December 2019
• Language Modeling with Deep Transformers
  K. Irie, A. Zeyer, R. Schlüter, H. Ney
  Interspeech, September 2019
  Best Student Paper Award.
  slides
• On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
  K. Irie, R. Prabhavalkar, A. Kannan, A. Bruguier, D. Rybach, P. Nguyen
  Interspeech, September 2019
  slides
• RWTH ASR Systems for LibriSpeech: Hybrid vs Attention
  C. Lüscher, E. Beck, K. Irie, M. Kitza, W. Michel, A. Zeyer, R. Schlüter, H. Ney
  Interspeech, September 2019
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
  Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, et al.
  Technical Report, Google, February 2019
• Investigation on Estimation of Sentence Probability by Combining Forward, Backward and Bi-directional LSTM-RNNs
  K. Irie, Z. Lei, L. Deng, R. Schlüter, H. Ney
  Interspeech, September 2018
• Improved training of end-to-end attention models for speech recognition
  A. Zeyer, K. Irie, R. Schlüter, H. Ney
  Interspeech, September 2018
• RADMM: Recurrent Adaptive Mixture Model with Applications to Domain Robust Language Modeling
  K. Irie, S. Kumar, M. Nirschl, H. Liao
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), April 2018
  slides
• Prediction of LSTM-RNN Full Context States as a Subtask for N-gram Feedforward Language Models
  K. Irie, Z. Lei, R. Schlüter, H. Ney
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), April 2018
  Best Student Paper Award; IEEE Spoken Language Processing Student Travel Grant Award.
• The 2016 RWTH Keyword Search System for Low-Resource Languages
  P. Golik, Z. Tüske, K. Irie, E. Beck, R. Schlüter, H. Ney
  International Conference on Speech and Computer (SPECOM), Lecture Notes in Computer Science, September 2017
• Investigations on byte-level convolutional neural networks for language modeling in low resource speech recognition
  K. Irie, P. Golik, R. Schlüter, H. Ney
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), March 2017
• LSTM, GRU, Highway and a Bit of Attention: An Empirical Overview for Language Modeling in Speech Recognition
  K. Irie, Z. Tüske, T. Alkhouli, R. Schlüter, H. Ney
  Interspeech, September 2016
• The RWTH/UPB/FORTH System Combination for the 4th CHiME Challenge Evaluation
  T. Menne, J. Heymann, A. Alexandridis, K. Irie, A. Zeyer, M. Kitza, P. Golik, I. Kulikov, L. Drude, R. Schlüter, H. Ney, R. Haeb-Umbach, A. Mouchtaris
  Interspeech Workshop on Speech Processing in Everyday Environments (CHiME), September 2016
• Automatic Speech Recognition Based on Neural Networks
  R. Schlüter, P. Doetsch, P. Golik, M. Kitza, T. Menne, K. Irie, Z. Tüske, A. Zeyer
  International Conference on Speech and Computer (SPECOM), Lecture Notes in Computer Science, August 2016
• Investigation on log-linear interpolation of multi-domain neural network language model
  Z. Tüske, K. Irie, R. Schlüter, H. Ney
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), March 2016
• Bag-of-Words Input for Long History Representation in Neural Network-based Language Models for Speech Recognition
  K. Irie, R. Schlüter, H. Ney
  Interspeech, September 2015
• On Efficient Training of Word Classes and Their Application to Recurrent Neural Network Language Models
  R. Botros, K. Irie, M. Sundermeyer, H. Ney
  Interspeech, September 2015
• The RWTH English lecture recognition system
  S. Wiesler, K. Irie, Z. Tüske, R. Schlüter, H. Ney
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), May 2014