The transcription of neuron-specific genes must be repressed in nonneuronal cells. REST/NRSF is a transcription factor that restricts the expression of many neuronal genes through interaction with the neuron-restrictive silencer element at the promoter level. PAHX-AP1 is a neuronal gene that is developmentally up-regulated in the adult mouse brain but that has no functional NRSE motif in its 5′ upstream sequence. Here, we report that the transcription factor AP4 and the corepressor geminin form a functional complex in which SMRT and histone deacetylase 3 are recruited. The functional complex represses PAHX-AP1 expression in nonneuronal cells and participates in regulating the developmental expression of PAHX-AP1 in the brain. This complex also serves as a transcriptional repressor of DYRK1A, a candidate gene for Down’s syndrome. Furthermore, compared with that in normal fetal brain, the expression of AP4 and geminin is reduced in Down’s syndrome fetal brain at 20 weeks of gestation age, at which time premature overexpression of dual-specificity tyrosine-phosphorylated and regulated kinase 1A (DYRK1A) is observed. Our findings indicate that AP4 and geminin act as a previously undescribed repressor complex distinct from REST/NRSF to negatively regulate the expression of target genes in nonneuronal cells and suggest that the AP4–geminin complex may contribute to suppressing the precocious expression of target genes in fetal brain.