Background The Four-Dimensional Symptom Questionnaire1 is a self-report questionnaire developed in the Netherlands to distinguish non-specific general distress from depression, anxiety, and somatization. This questionnaire is often used in different populations and settings and is being applied with different mediums (paper and pencil versus computerized).

Methods We used item response theory to investigate whether the 4DSQ measures the same construct (structural invariance) in the same way (scalar invariance) in two samples comprised of primary mental health care attendees: (i) clients who visited their General Practitioner responded to the 4DSQ by paper and pencil, and (ii) eHealth clients. Specifically, we investigated whether the distress items function differently in eHealth clients compared to General Practitioners’ clients and whether these differences lead to substantial differences at scale level.

Results Results showed that in general structural invariance holds for the distress scale. This means that the distress scale measures the same construct in both General Practitioners’ clients and eHealth clients. Furthermore, although eHealth clients have higher observed distress scores than General Practitioners’ clients, application of a multiple group generalized partial credit response model suggests that scalar invariance holds.

Conclusions Thus, the same cutoff scores can be used for classifying respondents as having low, moderate and high levels of distress in both settings.