Abstract Enhanced surface runoff in urban environments reduces water availability and limits the evaporative cooling potential. We evaluate surface runoff in 18 urban land surface models (ULSM) in Urban‐PLUMBER for 6,570 rainfall events across 21 urban sites. Surface runoff occurs when rainfall exceeds the infiltration, saturation, or interception capacity. Ten models omit at least one of these processes, while seven fail to increase runoff with increasing imperviousness. Surprisingly, some models lack any runoff during intense (>50mmh−1 ${50text{mmh} }^{-1}$) or prolonged (>20 mm) rainfall. Urban land surface models (ULSMs) turn 0%–86% rainfall into runoff. Most models produce runoff in agreement with an empirical comparison offered by the CN method, especially for high imperviousness. However, ULSM runoff exceeds CN runoff estimates for low impervious fraction, particularly by models with incomplete process description and for low‐intensity rainfall. The large discrepancy between ULSMs calls for advancing the urban hydrology representation in ULSMs, which is essential for correct simulation of evaporative cooling in cities.