Positive associations between vitamin D levels and hand grip strength have been reported worldwide, but the results are not consistent and few studies on East Asian populations have been published. The aim of this study was to determine whether such an association is present in a community-dwelling Japanese population, including elderly and middle-aged individuals. This study used a cross-sectional design. Participants were 492 community-dwelling individuals aged ≥ 40 years living in Yuzawa Town, Japan. The health check examination was conducted in 2015, and serum 25-hydroxyvitamin D [25(OH)D, an index of vitamin D levels], and hand grip strength were measured. Covariates were serum albumin concentration, body mass index, and physical activity level. The associations of serum 25(OH)D concentrations with hand grip strength and low grip strength (< 26 kg for men and < 18 kg for women) were analyzed using analysis of covariance and multiple logistic regression. Mean (standard deviation) age and serum 25(OH)D were 75.4 (9.0) years and 30.9 (9.1) ng/mL, respectively. The prevalence of serum 25(OH)D < 20, 20–29, and ≥ 30 ng/mL was 7.3%, 37.8%, and 54.9%, respectively. Mean hand grip strength in the 25(OH)D < 20 ng/mL group was significantly lower than that in the ≥ 30 ng/mL group (adjusted P ≤ 0.001). The 25(OH)D < 20 ng/mL group was significantly more likely to have low grip strength than the 25(OH)D ≥ 30 ng/mL group (odds ratio = 4.12). In conclusion, low serum 25(OH)D concentration (< 20 ng/mL) is associated with low grip strength in an older Japanese population.