Equitable access to the COVID-19 vaccine remains a public health priority. This study explores the association between ZIP Code-Tabulation Area level Social Vulnerability Indices (SVI) and COVID-19 vaccine coverage in Texas. A mixed-effects, multivariable, random-intercept negative binomial model was used to explore the association between ZIP Code-Tabulation Area level SVI and COVID-19 vaccination coverage stratified by the availability of a designated vaccine access site. Lower COVID-19 vaccine coverage was observed in ZIP codes with the highest overall SVIs (adjusted mean difference (aMD) = -13, 95% CI, -23.8 to -2.1, p < 0.01), socioeconomic characteristics theme (aMD = -16.6, 95% CI, -27.3 to -5.7, p = 0.01) and housing and transportation theme (aMD = -18.3, 95% CI, -29.6 to -7.1, p < 0.01) compared with the ZIP codes with the lowest SVI scores. The vaccine coverage was lower in ZIP Code-Tabulation Areas with higher median percentages of Hispanics (aMD = -3.3, 95% CI, -6.5 to -0.1, p = 0.04) and Blacks (aMD = -3.7, 95% CI, -6.4 to -1, p = 0.01). SVI negatively impacted COVID-19 vaccine coverage in Texas. Access to vaccine sites did not address disparities related to vaccine coverage among minority populations. These findings are relevant to guide the distribution of COVID-19 vaccines in regions with similar demographic and geospatial characteristics.