More than a (negative) Feeling: Validity of the Perceived Stress Scale in Serbian Clinical and Non-clinical Samples

The goal of the present study was to test the validity of a Serbian version of the Perceived Stress Scale. The PSS was administered to 157 psychiatric outpatients, 165 adults from the non-clinical population, and 283 university students. The results of the confirmatory factor analysis supported a bifactor model of the PSS with one general factor and two specific factors reflecting perceived distress and perceived self-efficacy. Internal consistencies of the scale and its two subscales were adequate across clinical and non-clinical samples. Results supported the ability of the scale to discriminate between clinical and non-clinical samples. The PSS evidenced good convergent validity, showing moderate to high positive correlations with measures of unpleasant emotional states and moderate negative correlations with positive affect and life satisfaction. All but one correlation remained significant after controlling for the measures of emotional distress. The results of the present research support validity of the PSS and its use in both clinical and non-clinical samples. and has since become the most frequently used scale for measuring perceived stress. The PSS was developed within the theoretical framework of the transactional model of stress (Lazarus, 1966), emphasizing the interaction between stressful events and individual appraisal of available coping resources (Lazarus & Folkman, 1984). According to this model, stress is a function of the discrepancy between perceived demands of the event and the individual's resources for meeting those demands. The original scale consisted of 14 items (PSS–14) aimed at assessing the degree to which individuals evaluate life circumstances and situations as stressful. As opposed to typical stress scales which list specific negative (and sometimes positive) life events, the PSS adopts a more general approach,

The Perceived Stress Scale was developed three decades ago (PSS; Cohen, Kamarch, & Mermelstein, 1983) and has since become the most frequently used scale for measuring perceived stress.The PSS was developed within the theoretical framework of the transactional model of stress (Lazarus, 1966), emphasizing the interaction between stressful events and individual appraisal of available coping resources (Lazarus & Folkman, 1984).According to this model, stress is a function of the discrepancy between perceived demands of the event and the individual's resources for meeting those demands.
The original scale consisted of 14 items (PSS-14) aimed at assessing the degree to which individuals evaluate life circumstances and situations as stressful.As opposed to typical stress scales which list specific negative (and sometimes positive) life events, the PSS adopts a more general approach, rendering it more sensitive to current life conditions and expectations about future life events (Cohen, 1986).The content of the PSS items encompasses perception of unpredictability, uncontrollability and sense of overload, which are the core aspects of stress (Cohen et al., 1983).
Besides the original 14-item scale, abridged scales have also been developed which contain four (PSS-4) and ten (PSS-10) items.The PSS-10 version has been most frequently used in the research, due to its brevity, simple administration, understandable items and favorable psychometric properties (Lee, 2012).The PSS-10 has been translated into more than 20 languages, including Chinese, German, Danish, Hebrew, Hungarian, Korean, Polish, Spanish and Urdu, to name just a few.The psychometric properties of the scale have been investigated in various samples, such as university students (Örücü & Demir, 2009;Roberti, Harrington, & Storch, 2006), general population (Andreou et al., 2011), adults with asthma (Sharp, Kimmel, Kee, Saltoun, & Chang, 2007), cardiac patients (Leung, Lam, & Chan, 2010), women with breast cancer (Golden-Kreutz, Browne, Frierson, & Anderson, 2004), depressed outpatients (Wongpakaran & Wongpakaran, 2010).Despite numerous translations and validity studies of the PSS, data regarding psychometric properties of the scale are still limited.Given that there has been considerable debate on the factor structure of the PSS since its development 30 years ago, most psychometric studies of the PSS were restricted to examining structural validity of the scale.Theoretically, the PSS is a unidimensional measure of perceived stress (Cohen & Williamson, 1988).However, the factor analytic studies have consistently suggested that perceived stress as measured with the PSS reflects a two-dimensional construct, comprised of perceived stress (helplessness, distress) and perceived self-efficacy (coping, counterstress) (e.g., Golden-Kreutz et al., 2004;Lehman, Burns, Gagen, & Mohr, 2012;Roberti et al., 2006).Surprisingly, although research findings favor a two-factor solution for the PSS and do not support the one-factor structure, most authors use this scale as a unidimensional measure of perceived stress and do not support using the two separate scores (e.g., Wang et al., 2011).Recently, it has been suggested that a bifactor model comprising a two-factor structure with a single underlying general factor of perceived stress best describes the structure of the PSS (Wu & Amtmann, 2013).A bifactor model of the PSS includes a general factor which influences all symptoms of perceived stress, and two separate factors of perceived stress and perceived self-efficacy that explain variance independently from the general factor.
Contrary to numerous studies investigating the factor structure of the PSS, there is a lack of data on the convergent validity of the scale and its unique relations with mental health indices.This is an important limitation of previous studies, having in mind that some authors argued that the PSS is "yet another measure of psychopathology or distress," (Lazarus, DeLongis, Folkman, & Gruen, 1985, p. 771) suggesting the problem of circularity and overlap between measure of perceived stress and emotional distress.Indeed, it is difficult to make a distinction between perceived stress and emotional distress, because they share a common thread of unpleasant emotions.The authors of the scale therefore included items which do not merely reflect the symptoms of psychological distress, typical for anxiety and affective disorders.For example, the items regarding perceived control and self-efficacy have also been included in the scale in order to minimize the overlap between the PSS and scales aimed at assessing emotional distress.It is expected that the PSS has moderate positive correlations with measures of mental health problems, sharing a significant amount of variance with depression and anxiety scales.Additionally, it is important to understand the conceptual difference between perceived stress as measured by the PSS and emotional distress.Perceived stress primarily refers to the cognitive evaluation of stress, i.e., "the degree to which situations in one's life are appraised as stressful" (Cohen et al., 1983, p. 387).On the other hand, emotional distress refers to mental health outcome variable, i.e., negative emotional consequence that may result from numerous factors other than perceived stress.Therefore, perceived stress does not necessarily result in negative outcomes (e.g., in highresilient people), while emotional distress is inherently negative.Previous studies have consistently shown that measures of perceived stress and emotional distress are closely linked and cannot be fully independent.For example, the PSS has substantial associations with measures of depression, anxiety and mental health problems, with typical correlation coefficients of approximately .60 (e.g., Andreou et al., 2011;Chaaya, Osman, Naassan, & Mahfoud, 2010;Remor, 2006;Wang et al., 2011).However, most of the validity studies failed to test the unique relations between the PSS and other mental health measures, after controlling for emotional distress and negative affect.Initial studies showed that perceived stress as measured with the PSS remains a significant predictor of mental health outcomes and physical symptoms after controlling for symptoms of psychological distress (e.g., Cohen, 1986).Nevertheless, subsequent studies have rarely examined to what extent the PSS overlaps with distress scales and whether the PSS is redundant.In addition, data on the psychometric properties of the PSS in clinical samples are still limited.
The goal of the present study was to examine the psychometric properties of a Serbian translation of the PSS in two non-clinical samples (undergraduate students and adults) and one clinical sample (psychiatric outpatients).Specifically, we assessed: a) the structural validity of the scale by means of confirmatory factor analysis; b) internal consistency reliability; c) convergent validity of the scale by examining correlations with measures of emotional distress (depression, anxiety, stress) and subjective well-being (life satisfaction, positive and negative affect).Furthermore, we examined the ability of the scale to discriminate between clinical and non-clinical samples, as well as the unique relationships between the PSS and measures of distress and well-being after partialling out the effects of unpleasant emotional states.

Samples and procedure
One clinical sample and two non-clinical samples (adult and student) were used in the present research.Participation in the study was voluntary and anonymity was assured.
The second non-clinical student sample included 283 students (146 females and 137 males) from the University of Novi Sad in Serbia, with a mean age 21.82 (SD = 1.98).Approximately 54% of participants were involved in a romantic relationship.

Instruments
Perceived Stress Scale-10 (PSS-10; Cohen et al., 1983) consists of 10 items rated on a 5-point scale, ranging from 0 (never) to 4 (very often).Participants were asked to rate how often they have felt and thought a certain way within the past month.The PSS includes six negatively worded items (e.g."In the last month, how often have you felt nervous and "stressed"?") and four positively worded items (e.g."In the last month, how often have you felt that things were going your way?").The PSS was translated into Serbian by the authors of the study, after obtaining permission of the PSS authors.The minor differences in translation were thoroughly discussed and resolved by consensus.The consensual version of the translated PSS was back-translated by the independent English language teacher.The items in Serbian and the back-translation were sent to one of the original authors (Dr.Sheldon Cohen) who approved the translation.The Serbian version of the PSS can be found in Appendix.
Depression Anxiety and Stress Scales (DASS-21;Lovibond & Lovibond, 1995) was used to assess negative emotional states.The DASS-21 consists of three 7-item scales: Depression (e.g., "I felt down-hearted and blue"), Anxiety (e.g., "I felt scared without any good reason") and Stress (e.g., "I felt that I was rather touchy").Respondents indicate the extent to which they agree with each statement on a 4-point scale, from 0 (did not apply to me at all) to 3 (applied to me very much, or most of the time).The DASS-21 has been widely used and showed good psychometric properties in clinical (e.g., Page, Hooke, & Morrison, 2007) and non-clinical samples (e.g., Henry & Crawford, 2005;Jovanović, Gavrilov-Jerković, Žuljević, & Brdarić, 2014).In the present sample, Cronbach's alphas for the Depression, Anxiety and Stress subscales were .82,.73,.79 in the student sample, .87,.83,.89 in the nonclinical adult sample, and .90,.86,.88 in the clinical sample, respectively.
The Serbian Inventory of Affect based on the Positive and Negative Affect Schedule-X (SIAB-PANAS; Novović & Mihić, 2008) is a Serbian translation and adaptation of the Positive and Negative Affect Schedule-X (PANAS-X; Watson & Clark, 1994).In the current study, we used the short form composed of two 10-items scales: Positive affect (PA) and Negative affect (NA).Participants were asked to report how they felt during the past month, using a 5-point Likert-type scale.In the present study, Cronbach's alphas for the PA subscale were .81(student sample), .91 (non-clinical adult sample) and .89(clinical sample) and .83,.87,.90 for the NA subscale, respectively.
Satisfaction With Life Scale (SWLS; Diener, Emmons, Larsen, & Griffin, 1985) was used to assess life satisfaction.The responses to each of the five items (e.g., "In most ways my life is close to my ideal") range from 1 (strongly disagree) to 7 (strongly agree).The SWLS showed favourable psychometric properties across various samples and cultures (e.g., Pavot & Diener, 2008).In the present study, Cronbach's alphas in the student sample, the nonclinical adult sample, and the clinical sample were .79,.91,.88,respectively.

Statistical analysis
Descriptive analysis, reliability and validity analyses were performed using the SPSS version 20.0.Confirmatory factor analysis (CFA) was performed using AMOS version 16.0.The maximum likelihood method conducted on correlation matrix was used.Several fit indices were computed to evaluate the models.Chi square (χ 2 ), chi square to degrees of freedom ratio (χ 2 /df), the Root Mean Square Error of Approximation (RMSEA) and Goodness of Fit index (GFI) were calculated as absolute fit statistics.Comparative Fit Index (CFI) and Bentler-Bonett Normed Fit index (NFI) were used as incremental fit indices.The χ 2 value should be nonsignificant to indicate a good fit, a χ 2 /df less than 3 (Kline, 2005), RMSEA values from .08 or less, and CFI, GFI and NFI values above .90are considered to indicate an acceptable fit (Hu & Bentler, 1999).Three competing models were examined via the CFA: a) the one-factor model with all ten items loading on a single factor; b) the two-factor model with two correlated factors: the Positive factor including four positively framed items, and the Negative factor covering six negatively framed items; c) the bifactor model with a general factor accounting for the commonality shared by the items, and two specific (orthogonal) factors reflecting the unique variance not accounted for by the general factor.The bifactor model allowed us to test whether the PSS was a general measure of perceived stress with two specific underlying dimensions.
Convergent validity was assessed by zero-order correlations and partial correlations (controlling for distress measures) between the PSS and measures of depression, anxiety, stress, and subjective well-being (life satisfaction, positive and negative affect).

Descriptive statistics, gender differences, and differences between clinical and non-clinical samples
The mean scores, standard deviations, and internal consistency coefficients for the PSS and its two subscales are shown for the whole samples, and separately for men and women across three samples, in Table 2.There were no significant gender differences in the PSS total score and the Positive subscale across three samples.Results indicated that women scored significantly higher than men on the Negative subscale in the student sample (Cohen's d = .30)and the non-clinical adult sample (Cohen's d = .31).
The PSS scores were compared across three samples using one way ANOVAs.There were significant differences between groups for the PSS total score [F(2, 602) = 62.68, p <.001, η 2 = .17],the Positive subscale [F(2, 602) = 30.12,p <.001, η 2 = .09]and the Negative subscale [F(2, 602) = 61.79,p <.001, η 2 = .17].Tukey's HSD post-hoc test showed that participants from the clinical sample scored significantly higher that participants from two non-clinical samples (adults and students) on the PSS full scale (Cohen's d = .85and 1.07, respectively) and the Negative subscale (Cohen's d = .84 and 1.06,respectively), and significantly lower on the Positive subscale (Cohen's d = .63 and .73,respectively).Students and the non-clinical adult sample did not differ on the PSS full scale and the Positive subscale, while adults scored significantly higher on the Negative subscale than students (Cohen's d = .25).

Reliability and interscale correlations
The internal consistency reliability of the PSS and two subscales was found to be adequate across the three samples (all α's ≥ .70,except the α for the Positive subscale among the female students, which was .69).Reliability of the Positive subscale was somewhat lower than the reliability of the PSS full scale and the Negative subscale, but can be considered adequate, given the small number of items (four).
The correlations between the Positive subscale and the Negative subscale of the PSS were as follows: -.45 (student sample), -.61 (adult sample), -.53 (clinical sample).These low to moderate correlations suggest that the two subscales represent two related yet distinct components of perceived stress.

Convergent validity
Convergent validity of the full PSS and its two subscales (see Table 3 and  Table 4) was examined via correlational analyses with measures of unpleasant emotional states (depression, anxiety, stress) and subjective well-being (satisfaction with life, positive and negative affect).Partial correlations between the PSS and other measures controlling the influence of negative affect have also been calculated, in order to test the unique associations between perceived stress and convergent measures.Note: all correlations are significant at p <.01.Pos.= Positive subscale, Neg.= Negative subscale.
The full PSS showed moderate to high positive correlations with measures of depression, anxiety, stress and negative affect, and moderate negative correlations with measures of positive affect and satisfaction with life.It is important to note that the correlations between the PSS and measures of distress (but not life satisfaction and positive affect) were consistently lower1 in student sample, as compared to adult samples (both clinical and non-clinical).All but one correlation (between the PSS and DASS-21 Stress subscale in clinical sample) remained significant when negative affect was partialed out, suggesting that the PSS is not redundant with other measures of distress and well-being.The same results were observed when effects of depression, anxiety and stress (DASS-21) were controlled in partial correlation analysis (data not shown).Compared with the Positive subscale, the correlations of the Negative subscale were consistently stronger with measures of unpleasant emotional states across all three samples (Fisher's r to z transformation indicated that there were significant differences between the correlations in 9 out of 12 cases).Two subscales did not differ with regard to their relations with life satisfaction and positive affect (all z's are nonsignificant).

Discussion
The aim of the present study was to evaluate the psychometric properties (construct validity, internal consistency and convergent validity) of a Serbian translation of the 10-item Perceived Stress Scale (PSS).Additionally, the ability of the scale to discriminate between clinical and non-clinical samples was tested.The psychometric properties were investigated in two non-clinical samples: students (N = 283) and adults (N = 165), and one clinical sample (N = 157).In general, the results lend preliminary support to the validity and the reliability of the PSS in a Serbian samples.
The construct validity of the PSS was examined by means of confirmatory factor analysis.A bifactor model with one general factor reflecting the overlap across all ten items and two specific factors representing perceived distress (negatively worded items) and perceived self-efficacy (positively worded items) demonstrated the best fit for the data.This finding suggests that the PSS includes the general factor of perceived stress, along with two separate factors.The bifactor model supported in the present study indicated that both the total score and the subscale scores of the PSS can be used.These findings are consistent with the results of previous studies that used both exploratory factor analysis and confirmatory factor analysis to test the construct validity of the PSS-10 across various samples (e.g., Andreou et al., 2011;Chaaya et al., 2010;Leung et al., 2010;Ng, 2013;Roberti et al., 2006).
The reliability estimates of the full PSS and its two subscales are consistent with previous research and indicate adequate internal consistencies.The PSS scales met the cut-off of α > .70(Nunnally & Bernstein, 1994) across all samples, except for the Positive subscale in the student female sample, which was below the recommended threshold (α = .69).
No significant gender differences were found in the full PSS and the Positive subscale across the three samples.Women scored significantly higher on the Negative subscale in the two non-clinical samples (Cohen's d values indicated small effect size), but not in the clinical sample.These results accord with previous research, yielding inconsistent findings on gender differences in the PSS scores across various samples (Lee, 2012).Although most previous studies showed that women report greater perceived stress than men (e.g., Andreou et al., 2011;Cohen & Janicki-Deverts, 2012;Örücü & Demir, 2009), some studies did not find significant gender differences (e.g., Ramirez & Hernandez, 2007;Roberti et al., 2006).
As expected, there were significant differences between the clinical sample and non-clinical samples in the PSS scores, with individuals from the clinical sample showing higher level of perceived stress.This finding is consistent with previous research, showing that higher levels of depression and anxiety are associated with higher perceived stress (e.g., Bergdahl & Bergdahl, 2002;Wongpakaran & Wongpakaran, 2010) and that PSS scores were higher in clinical samples relative to the community sample (Lavoie & Douglas, 2012).Large differences between clinical and non-clinical samples in the PSS scores observed in the present study (as indicated by the Cohen's d effect size statistic) suggest that this scale discriminates well between clinical and non-clinical populations and that it can be used as an outcome measure.
The PSS demonstrated good convergent validity.The full scale and its two subscales showed significant and substantial correlations with measures of distress and subjective well-being.The PSS showed moderate to high positive correlation with negative emotional states, in accordance with previous studies (e.g., Remor, 2006;Wang et al., 2011).Close relationships between the PSS and the DASS-21 are expected, given that symptoms of depression, anxiety, tension and perceived stress share a common thread of negative affect and tend to overlap considerably (e.g., Bener, Al-Kazaz, Ftouni, Al-Harthy, & Dafeeah, 2013).The size of correlation coefficients and the results of partial correlation analysis suggest that perceived stress is closely associated with, yet distinct from measures of emotional distress.The PSS showed unique associations with positive and negative measures of well-being after controlling for influence of other unpleasant emotional states (depression, anxiety, stress and negative affect).These results suggest that the PSS is not a merely measure of distress, but a distinct measure of perceived stress, not redundant with negative emotions.It is important to note that the associations of the PSS with measures of unpleasant emotional states differed between the three samples.The correlations within the student sample were consistently lower than within the two adult samples.These results suggest that the PSS is more closely linked to negative emotions in the adult samples as compared to the student sample.Our findings warrant further research to clarify whether the observed differences are due to better coping skills in students which reduce the negative effects of perceived stress on the emotional distress.
Results regarding convergent validity of the two PSS subscales suggest that the use of the total score of the PSS may not be always justified and may lead to loss of important information.For example, the full PSS showed a similar pattern of correlations with measures of negative emotions to the Negative subscale, while the correlations with positive affect and life satisfaction were comparable to the Positive subscale.These results indicate that the PSS taps two somewhat different dimensions of perceived stress, which raises the questions of whether the total PSS score is useful and what this composite score actually represents.Future research should further examine whether the two PSS dimensions serve different functions in the stress process, lead to different outcomes and whether they are differently influenced by important mental health variables, such as positive/negative life events and positive expectations.
The existence of the two subscales within the PSS may be a result of the item wording, a method artifact often observed among the scales comprising both positive and negative items (Horan, DiStefano, & Motl, 2003).However, the content of the positively worded and negatively worded items in the PSS present conceptual differences.Positively worded items relate to perceived sense of control (e.g., item 7: In the last month, how often have you been able to control irritations in your life?) and perceived self-efficacy (e.g., item 4: In the last month, how often have you felt confident about your ability to handle your personal problems?),while the negatively worded items encompass both affective responses to stressful events (e.g, item 1: In the last month, how often have you been upset because of something that happened unexpectedly?) and a sense of feeling overwhelmed and unable to cope (e.g., item 6: In the last month, how often have you found that you could not cope with all the things that you had to do?).In terms of the cognitive-relational theory of stress (Lazarus & Folkman, 1984), it seems that the Positive subscale items assess secondary appraisals (evaluations of coping resources), while the Negative subscale items assess both immediate effects (negative affect) and secondary appraisals, which are related yet distinct aspects of stressful experience.We argue that more empirical data are needed to help researchers determine whether and when using the total score and/or the two subscale scores is justified.
The present study has some limitations.First, although participants were from both clinical and non-clinical populations, the sample size was relatively small and convenience samples were used.In order to generalize the finding, future research should use large random samples.Second, the cross-sectional design did not allow us to examine the prospective predictive validity of the PSS.Future studies should use longitudinal design to investigate whether the PSS as a global measure of perceived stress and its two components predict mental health outcomes prospectively.
To summarize, the results of the present study showed that the Serbian version of the PSS is a reliable and valid measure of perceived stress across both clinical and non-clinical samples.

Table 1
Goodness-of-fit indices of the confirmatory factor analysis Note: Sample size is 605 for total sample, 283 for student sample, 165 for adults sample and 157 for clinical sample; df = degrees of freedom; RMSEA = Root Mean Square Error of Approximation; CI = Confidence Intervals; CFI = Comparative Fit Index; GFI = Goodness of Fit Index; NFI = Normed Fit Index.

Table 3
Convergent validity of the full PSS Numbers in parentheses show partial correlations controlling for the influence of negative affect.All correlations are significant at p <.01, except the partial correlation between the PSS and DASS-21 Anxiety subscale in the clinical sample which is non-significant. Note:

Table 4
Convergent validity of the PSS subscales