The detection of small and subtle lesions in high-resolution 3D volumes is a highly relevant, yet far from solved task in biomedical imaging. We here address a specific task in detecting certain types of epileptogenic lesions through our novel semi-supervised spatially context aware transformer (SpyCAT) approach to anomaly detection. SpyCAT is modeling-based in the sense that it builds on specific assumptions that constitute what is normal and what constitues relevant deviations from normality. We explicitly use these assumptions to justify the inductive bias of our anomaly detection approach. The resulting SpyCAT system is patch-based and uses a transformer architecture to process discrete tokens obtained from a vector quantizing variational autoencoder, which produces counterfactual patches through full 3D convolutions of each patch. We evaluate our approach on the grounds of point-annotations of two subtypes of epileptogenic lesions, using validation measures that build on the Metrics Reloaded framework, showing that SpyCAT can reliably identify and localize the lesion types under consideration, and outperforms state-of-the-art reference methods.