Assessing the impact of planned social change

doi:10.1016/0149-7189(79)90048-X

Evaluation and Program Planning

Volume 2, Issue 1, 1979, Pages 67-90

https://doi.org/10.1016/0149-7189(79)90048-X Get rights and content

Abstract

It is a special characteristic of all modern societies that we consciously decide on and plan projects designed to improve our social systems. It is our universal predicament that our projects do not always have their intended effects. Very probably we all share in the experience that often we cannot tell whether the project had any impact at all, so complex is the flux of historical changes that would have been going on anyway, and so many are the other projects that might be expected to modify the same indicators.

It seems inevitable that in most countries this common set of problems, combined with the obvious relevance of social science research procedures, will have generated a methodology and methodological specialists focused on the problem of assessing the impact of planned social change. It is an assumption if this paper that, in spite of differences in the forms of government and approaches to social planning and problem-solving, much of this methodology can be usefully shared — that social project evaluation methodology is one of the fields of science that has enough universality to make scientific sharing mutually beneficial. As a part of this sharing, this paper reports on program impact assessment methodology as it is developing in the United States today.

References (124)

D.T. Campbell et al.
Making the case for randomized assignment to treatments by considering the alternatives: Six ways in which quasi-experimental evaluations in compensatory education tend to underestimate effects
J.K. Anderson
Evaluation as a process in social systems
(1973)
Scarvia B. Anderson et al.
Encyclopedia of educational evaluation
(1975)
Orley Ashenfelter
Estimating the effect of training programs on earnings
The Review of Economics and Statistics
(1978)
D.C. Baldus
Welfare as a loan: An empirical study of the recovery of public assistance payments in the United States
Stanford Law Review
(1973)
B.S. Barnow
The effects of Head Start and socioeconomic status on cognitive development of disadvantaged children
R.A. Bauman et al.
Working with complex data files: II. The Wisconsin assets and incomes studies archive
B. Beck
Cooking the welfare stew
H.M. Becker et al.
Making the grade
(1968)
H.W. Becker
Sociological work: Method and substance
(1970)

J.S. Berliner

Factory and manager in the U.S.S.R.

(1957)

P.M. Blau

The dynamics of bureaucracy

(1963)

R.F. Boruch et al.

Preserving confidentiality in evaluative social research: Intrafile and interfile data analysis

G.E.P. Box et al.

A change in level of non-stationary time series

Biometrika

(1965)

G.E.P. Box et al.

Intervention analysis with applications to economic and environmental problems

Journal of the American Statistical Association

(1975)

G.E.P. Box et al.

Time-series analysis: Forecasting and control

(1970)

D.T. Campbell

Pattern matching as an essential in distal knowing

D.T. Campbell

Administrative experimentation, institutional records, and nonreactive measures

Donald T. Campbell

A phenomenology of the other one: Corrigible, hypothetical and critical

D.T. Campbell

Reforms as experiments

American psychologist

(1969)

D.T. Campbell

Considering the case against experimental evaluations of social innovations

Administrative Science Quarterly

(1970)

D.T. Campbell

Methods for an experimenting society

D.T. Campbell

Experimentation revisited: A conversation with Donald T. Campbell

Evaluation

(1973)

D.T. Campbell

Qualitative knowing in action research

D.T. Campbell et al.

Confidentiality-preserving modes of access to files and to interfile exchange for useful statistical analysis

Evaluation Quarterly

(1977)

D.T. Campbell et al.

How regression artifacts in quasi-experimental evaluations can mistakenly make compensatory education look harmful

D.T. Campbell et al.

Convergent and discriminant validation by the niultitrait-multimethod matrix

Psychological Bulletin

(1959)

D.T. Campbell et al.

Direction-of-wording effects in the relationsliips between scales

Psychological Bulletin

(1967)

D.T. Campbell et al.

Experimental and quasi-experimental designs for research

(1966)

V.G. Cicirelli

The impact of Head Start: An evaluation of the effects of Head Start on children's cognitive and affective development

Ross F. Conner

Selecting a control group: An analysis of the randomization process in twelve social reform programs

Evaluation Quarterly

(1977)

T.D. Cook et al.

Sesame Street revisited: A case study in evaluation research

(1975)

T.D. Cook et al.

The design and conduct of quasi-cxperiments and true experiments in field settings

T.D. Cook et al.

Quasi-experimentation: Design and analysis issues for field settings

(1979)

L.J. Cronbach

Response sets and test validity

Evaluation and Psychological Measurement

(1946)

L.J. Cronbach

Further evidence on response sets and test design

Educational and Psychological Measurement

(1950)

H.P. David

Family planning and abortion in the socialist conntires of Central and Eastern Europe

(1970)

H.P. David et al.

Abortion legislation. The Romanian experience

Studies in Family Planning

(1971)

S.M. Director

Underadjustment bias in the evaluation of manpower training

Evaluation Quarterly

(1979)

J.D. Douglas

The social meanings of suicide

(1967)

A.L. Edwards

The social desirability variable in personality assessment and research

(1957)

G.W. Fairweather

Methods for experimental social innovation

(1967)

G.W. Fairweather et al.

Experimental methods for social policy research

(1977)

J.L. Fischer

The uses of Internal Revenue Service data

Cited by (329)

AI for the people? Embedding AI ethics in HR and people analytics projects
2024, Technology in Society
Technological advances and the “zeitgeist” led to increased use of AI in HR Analytics. Using machine learning models in this field might be beneficial but requires some unique considerations in how organizations govern, assure, audit, and support the use of AI. While substantial work has been done on AI ethics, organizational implantation studies are rare and often anecdotal, particularly in HR. In this article, we would like to better understand the common ethical considerations and organizational responses for developing AI for HR use and how different organizations engage with them by comparing two case studies.
This paper begins by reviewing AI ethical issues specific to HR and describes the common mitigation strategies proposed in the literature. After this introduction, case studies are illustrated, and the organizational responses in both cases are analyzed. The discussion section shows that although the organizations were aware of the AI ethics issues, their responses varied and depended on different governance and self-regulation mechanisms that might have impinged their ability to proactively engage with ethical issues. A second finding is that the organization used only a relatively small set of mitigation strategies proposed in the literature, and we propose several reasons for this. These insights might be important to various organizations deploying or developing AI for HR use that want better governance, assurance, and audit mechanisms to responsibly address the ethical issues raised by AI in this field.
Building less-flawed metrics: Understanding and creating better measurement and incentive systems
2023, Patterns
Metrics are useful for measuring systems and motivating behaviors in academia as well as in public policy, medicine, business, and other systems. Unfortunately, naive application of metrics to a system can distort the system and even undermine the original goal. There are two interrelated problems to overcome in building better metrics in academia and elsewhere. The first, specifying evaluable metrics that correspond to the goals, is well recognized but still often ignored. The second, minimizing perverse effects that undermine the metric or that enable people to game the rewards, is less recognized but is critical. This perspective discusses designing metrics, beginning with design considerations and processes; the presentation of specific strategies for mitigating perverse impacts, including secrecy, randomization, diversification, and post hoc specification; and continuing with important desiderata and tradeoffs involved with examples of how they can complement each other or differ. Finally, this perspective presents a comprehensive process integrating these ideas.
Bridging Reasoning and Results: An Assessment Validation Perspective on the PhD in Design
2023, She Ji
This article uses assessment validation as a framework to construct a theory of how to assess a PhD in design. This also has implications for what the PhD degree should be in design. When examiners assess a PhD student’s dissertation or defense, interpretation arguments and use arguments connect the assessment to the ultimate purpose and consequence of the PhD project they assess. My conclusion is that this argument-based approach strengthens the credibility, meaningfulness, and fairness of PhD assessment. The perspective of assessment validation enhances the PhD in design by focusing validation on inferences and assumptions that connect scoring, generalization, extrapolation, and the implications of PhD assessment.
Government funding incentives and study program capacities in public universities: Theory and evidence
2024, Oxford Economic Papers
Attribution of Work in Programming Teams with Git Reporter
2024, SIGCSE 2024 - Proceedings of the 55th ACM Technical Symposium on Computer Science Education
Integrity and the University
2024, Philosophy of Management

View all citing articles on Scopus

^☆: Reprinted, with minor revisions and additions, from Gene M. Lyons, (Ed.), Social Research and Public Policies, Hanover, New Hampshire: University Press of New England, 1975, with the permission of the publisher. Preparation of this paper has been supported in part by grants from the Russell Sage Foundation and the National Science Foundation Grants Numbers GSOC-7103704-03 and BNS76-23920. An earlier version of this paper was presented to the Visegrad, Hungary, Conference on Social Psychology, May 5–10, 1974.

View full text

Assessing the impact of planned social change☆

Abstract

Evaluation as a process in social systems

Encyclopedia of educational evaluation

Estimating the effect of training programs on earnings

The Review of Economics and Statistics

Welfare as a loan: An empirical study of the recovery of public assistance payments in the United States

Stanford Law Review

The effects of Head Start and socioeconomic status on cognitive development of disadvantaged children

Working with complex data files: II. The Wisconsin assets and incomes studies archive

Cooking the welfare stew

Making the grade

Sociological work: Method and substance

Factory and manager in the U.S.S.R.

The dynamics of bureaucracy

Preserving confidentiality in evaluative social research: Intrafile and interfile data analysis

A change in level of non-stationary time series

Biometrika

Intervention analysis with applications to economic and environmental problems

Journal of the American Statistical Association

Time-series analysis: Forecasting and control

Pattern matching as an essential in distal knowing

Administrative experimentation, institutional records, and nonreactive measures

A phenomenology of the other one: Corrigible, hypothetical and critical

Reforms as experiments

American psychologist

Considering the case against experimental evaluations of social innovations

Administrative Science Quarterly

Methods for an experimenting society

Experimentation revisited: A conversation with Donald T. Campbell

Evaluation

Qualitative knowing in action research

Confidentiality-preserving modes of access to files and to interfile exchange for useful statistical analysis

Evaluation Quarterly

How regression artifacts in quasi-experimental evaluations can mistakenly make compensatory education look harmful

Convergent and discriminant validation by the niultitrait-multimethod matrix

Psychological Bulletin

Direction-of-wording effects in the relationsliips between scales

Psychological Bulletin

Experimental and quasi-experimental designs for research

The impact of Head Start: An evaluation of the effects of Head Start on children's cognitive and affective development

Selecting a control group: An analysis of the randomization process in twelve social reform programs

Evaluation Quarterly

Sesame Street revisited: A case study in evaluation research

The design and conduct of quasi-cxperiments and true experiments in field settings

Quasi-experimentation: Design and analysis issues for field settings

Response sets and test validity

Evaluation and Psychological Measurement

Further evidence on response sets and test design

Educational and Psychological Measurement

Family planning and abortion in the socialist conntires of Central and Eastern Europe

Abortion legislation. The Romanian experience

Studies in Family Planning

Underadjustment bias in the evaluation of manpower training

Evaluation Quarterly

The social meanings of suicide

The social desirability variable in personality assessment and research

Methods for experimental social innovation

Experimental methods for social policy research

The uses of Internal Revenue Service data