Fossil fuel energy infrastructure poses health risks for local communities, primarily due to the presence of air pollution emissions and other hazards. There is also evidence of racial/ethnic disparities in the siting of this infrastructure for select components. However, population counts and demographic composition near fossil fuel energy infrastructure have not been systematically characterized across all types, supply chain stages, and predominant fuel types. Here, we construct a dataset of 25 elements of fossil fuel energy infrastructure and characterize the populations living near this infrastructure (defined as within 800 m [∼0.5 mile] or 1.6 km [∼1 mile]). We estimated that 46.6 million people in the contiguous U.S., representing 14.1% of the population, live within 1.6 km of at least one piece of energy infrastructure, with racial/ethnic disparities observed across nearly all stages of the supply chain. End use infrastructure has the most people residing within 1.6 km, with 20.9 million people, followed by extraction (20.3 million), and storage (6.16 million). Storage infrastructure has an average of ∼2,900 people living within 1.6 km of each element; end use infrastructure has an average of 1,900 people residing within 1.6 km of each element; extraction infrastructure has an average of 17 people residing within 1.6 km of each element. Almost 90% of the population near end use, transportation, refining, and storage infrastructure are in urban areas. Our results represent a substantial population in the U.S. that is potentially exposed to hazards that are not well-characterized, with unknown cumulative impacts, and which constitute a major environmental justice issue.