The purpose of this study was to describe and explain Japanese generalist elementary teachers' experiences teaching physical education (PE) to Japanese language learner (JLL) children in public schools in Japan. Participants were seven public elementary school teachers (two females and five males) who had experience teaching JLL children. Data sources included a demographic questionnaire, online semi-structured interviews, and follow-up email interviews. Framed in positioning theory, three themes were identified from the data analysis: (a) seeking PE lesson ideas from observing children's performance in other academic subjects, (b) learning how to interact with parents of JLL children, and (c) accommodating JLL children's cultural and behavioral differences in PE. The findings indicate that all seven Japanese public elementary teachers experienced a wide range of physical, emotional, and social struggles when they taught JLL children in PE class. This study suggested that PE has the potential to enrich teachers' and children's cultural awareness and mutual understanding and as a tool to raise awareness of social justice and diversity.