Investigating the organization of trait aggression and impulsivity in the prefrontal cortex (PFC) advances our understanding of the neuropsychobiology of self-control. While the orbital aspect of the PFC (OFC) has received attention, there is reason to believe the lateral aspect is also relevant. In the current study using magnetic resonance imaging, gray matter volumes in lateral PFC (LPFC) were derived in a heterogeneous male psychiatric sample (N=36) in which OFC volumes had previously been reported. In an analysis using self-report measures of trait impulsivity and aggression, the left LPFC accounted for significant variance in attentional aspects of impulsivity (13%) and aggression (10%) but not motor aspects of impulsivity, as hypothesized. The OFC was associated with motor impulsivity (left-20%; right-14%) and was also more robustly associated with aggression (left-36%; right-16%). A social/emotional information processing model was explored, based upon whether the LPFC or the OFC depended upon one another for their association to trait aggression and impulsivity. It was demonstrated that association of the LPFC to both aggression and attentional impulsivity depended upon the OFC, while the converse was not supported. The LPFC appears relevant to the higher-order aspects of a cortical self-control network, and that relevance is dependent upon the robust contribution of the OFC.