Background: People infected with Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) experience a wide range of clinical manifestations, from asymptomatic and mild illness to severe illness and death, influenced by age and a variety of comorbidities. Neutralizing antibodies (nAbs) are thought to be a primary immune defense against the virus. Large, diverse, well-characterized cohorts of convalescent individuals provide standardized values to benchmark nAb responses to past SARS-CoV-2 infection and define potentially protective levels of immunity. Methods and findings: This analysis comprises an observational cohort of 329 HIV-seronegative adults in the United States (n = 167) and Peru (n = 162) convalescing from SARS-CoV-2 infection from May through October 2020. The mean age was 48 years (range 18 to 86), 54% of the cohort overall was Hispanic, and 34% identified as White. nAb titers were measured in serum by SARS-CoV-2.D614G Spike-pseudotyped virus infection of 293T/ACE2 cells. Multiple linear regression was applied to define associations between nAb titers and demographic variables, disease severity and time from infection or disease onset, and comorbidities within and across US and Peruvian cohorts over time. nAb titers peaked 28 to 42 days post-diagnosis and were higher in participants with a history of severe Coronavirus Disease 2019 (COVID-19) (p < 0.001). Diabetes, age >55 years, male sex assigned at birth, and, in some cases, body mass index were also independently associated with higher nAb titers, whereas hypertension was independently associated with lower nAb titers. nAb titers did not differ by race, underlying pulmonary disease or smoking. Two months post-enrollment, nAb ID50 (ID80) titers declined 3.5 (2.8)-fold overall. Study limitations in this observational, convalescent cohort include survivorship bias and missing early viral loads and acute immune responses to correlate with the convalescent responses we observed. Conclusions: In summary, in our cohort, nAb titers after SARS-CoV-2 infection peaked approximately 1 month post-diagnosis and varied by age, sex assigned at birth, disease severity, and underlying comorbidities. Our data show great heterogeneity in nAb responses among people with recent COVID-19, highlighting the challenges of interpreting natural history studies and gauging responses to vaccines and therapeutics among people with recent infection. Our observations illuminate potential correlations of demographic and clinical characteristics with nAb responses, a key element for protection from COVID-19, thus informing development and implementation of preventative and therapeutic strategies globally.