Category Archives: Methodology

Avocascience? Or Objective Neutrality?

Posted on by
1

From a recent post by Jay Ulfelder,

“When you hear the term “conflict of interest,” you probably think of corporations paying for studies that advance their commercial interests. I know I do. It’s easy to see why studies on the effectiveness of new drug therapies or the link between pollution and cancer, for example, warrant closer scrutiny when they’re funded by firms with profits riding on the results. You don’t have to be a misanthrope to believe that the profit motive might have shaped the analysis, and there are enough examples of outright fraud to make skepticism the prudent default setting.

That’s not the only conflict that can arise, though. What I think many scholars working in comparative politics don’t appreciate as much as we should is that it’s also possible for political values and advocacy to play a similar role, and to similar effect. When a researcher’s work deals with issues on which he or she has strong moral beliefs, that confluence can hinder his or her ability to identify and fairly weigh relevant evidence. Confirmation bias is hard to overcome, especially in studies that rely entirely on an author’s interpretation, as many qualitative studies do. The problem is even more intense if the researchers’ personal life is interwoven with her work. Certain conclusions may be more palatable or appealing to people with certain values, and it can be professionally and personally damaging for researchers to report findings that suggest the work their friends and colleagues are doing may not be all that useful, or may even be counterproductive.”

I think this is a larger problem than people realize.  We all have certain moral and conceptual beliefs that we carry to make sense of the world and it is very difficult to separate those beliefs from our research and from our assessments of other people’s research.  In that sense, I think we need to be more upfront about these realities and the beliefs that people carry with them.
Continue reading →

In my primary research field, Aboriginal-settler relations in Canada, this is a particularly tricky problem, given that Indigenous peoples have been so disempowered and impoverished by the Crown.  There’s a real temptation to use one’s scholarship and peer review role as advocacy.  I know I certainly struggle with this issue.

As an author, I think the solution is to be upfront about your beliefs.  Recently, I’ve been reading a bunch books on Indigenous methodology and one of the many things that they do right is to announce early on who they are, where they come from, and what kind of perspective they bring to the table.  Rather than pretending to be “objective”, which quite frankly, is an impossibility in my view, we as authors should be up front about our beliefs so the readers know where we are coming from. And reviewers should assess manuscripts by respecting those beliefs rather than rejecting ideas outright because they don’t gel with our own beliefs.

Some of my work, for instance, uses rational choice to analyze treaty and devolution negotiations. During the peer review process, I’ve encountered multiple reviewers who have rejected my research outright because they don’t like rational choice.  Rarely do they ever say why the use of rational choice is inappropriate to the case at hand or how the evidence does not support the argument.

As a peer reviewer, I try to approach new research by accepting the theoretical choices of the author as a given (at least at first). So if an author decides to use political culture, for instance, which is a concept I’m highly skeptical of, I initially accept that choice and ask: a) why is this concept better than others for explaining the phenomenon at hand? b) how well does the author sketch out, deploy, and defend the concept/argument with evidence?

So what are my beliefs? Quite frankly, I think I’m more confused and uncertain about the world than anything else! My scholarship has been characterized as right wing, left wing, moderate, and libertarian, all at the same time by different people. I hope that reflects my commitment to being open to the very real possibility that my past and present views about the world are wrong (or maybe I’m just engaging in Bayesian updating!).

My Final Verdict on the Flipped Classroom

Posted on by
Reply

Late last week, I finally finished grading the last assignments and exams for my first year seminar on Politics and Film and submitted final grades to the Registrar’s office.  Whew! What a term!

As I’ve blogged about previously, this year I’ve been using the flipped classroom technique to teach my first year seminar course on Understanding Power and Conflict through Film. The main idea behind this pedagogy is to use class time more effectively.  Rather than the professor lecturing or facilitating discussion for the entire class, as is the norm for most first year classes and upper year seminars respectively, the flipped classroom pedagogy asks students to learn the basics at home and to spend actual class time applying their learning to a variety of problems and situations, with the guidance of the instructor.

In my first year seminar this term, for instance, students were asked to read several academic readings at home each week before completing an online quiz that required them to find examples from the real-world to illustrate certain ideas and concepts from the readings.  From those quizzes, I would write a 10-15 minute lecture on the topic for that week, spending particular time going over the concepts that the students seemed to have the most trouble with in the quiz.  After that short lecture, we would spend the rest of the class in small and large groups working on applying the various concepts to a variety of problems and situations. We would usually end with a class discussion or formal debate on the strengths and weaknesses of the concepts or theories being studied that week.

In previous blogs, I talked about how I used the flipped classroom pedagogy to structure my lessons on the state of nature and rational choice/game theory.  For the rest of the term, we covered the concepts of structure and agency, institutions, class and capitalism, and colonialism.  The basic structure for these classes was the same as outlined above. The final exam asked students to watch a film and explain what happened in that film by using the course concepts and readings covered in class during the term.

So what’s the final verdict? I think the flipped classroom pedagogy is a keeper.  Students seemed to like the the class and format.  In an anonymous survey, for instance, students mentioned they learned more in this class than any other first year class they took this term.  They liked how the class exercises, discussions, movies, and assignments, were all focused on helping students learn and apply course concepts to a wide variety of phenomenon. While some first year seminars this term suffered from high levels of student attrition (e.g. one instructor reported that s/he had only eight regularly attending students by the end of the course), my classes always had between 19 or 20 students out of a total 20 in attendance.  While another first year seminar instructor complained that the 3 hour length of their seminar was far too long, my classes always went the full three hours and none of my students ever complained about this fact; indeed, some (just a few though!) were disappointed when class exercises and discussion had to be cut off due to the 3 hour class limit!

That being said, there are some things about the flipped classroom that I need to tinker with next year:

  • Online quizzes: Although they were really useful for helping me figure out what topics and concepts the students were struggling with (and hence what I needed to cover in lecture), they were really time consuming to grade.
  • Small group work: Most small groups had fabulous discussions.  However, some small groups spoke very little so I need to find stronger incentives for individuals to participate in small group discussions.
  • More variety in terms of in-class activities.  The state of nature and game theory weeks were really easy to do, partly because they were the kind of topics that were most open to games.  The weeks after, were slightly more difficult.  I ended up creating formal debates at the end of most classes, which involved two debating teams trying to convince a panel of their peers that their position was the more convincing one.  Most students liked these formal debates. A few did not.
  • Modify my goals and expectations.  For the most part, I assigned academic readings on what are some pretty heavy social science concepts.  Unsurprisingly, perhaps, students did not always come away with a full mastery of the concepts but they did learn some core ideas about them and how they might be applied to a variety of situations.  At first, I thought this meant I had failed in achieving my course learning objectives.  But by the end of the course, I saw this outcome as a victory, especially once I thought about how these outcomes compared to the typical final exam answers I get in my second year lecture course.

There were other issues as well, for this course, which I will need to tinker with but these other issues had more to do with non-flipped classroom things.

I’ve been told that I will be allowed to teach this course next year, which I am really excited about doing!  The next task, however, will be to see if I can “flip” my second year lecture course on Canadian politics, enrollment of 125.  Stay tuned!

Genes, Politics, and Epigenetics: Wonderfully Messy Complexity

Posted on by
Reply

Following on Chris’s theme of genes and politics, I want to remark on a contrast in his post between the approaches political scientists have adopted from behavioural and population genetics, on the one hand, and findings on the health benefits of vitamin D, on the other. Those findings have to do with gene regulation, and are part of the burgeoning field of epigenetics.

This field has longstanding roots in older work on transposons, noncoding RNA, and various mechanisms regulating genetic expression and development and evolution, the deeper implications of which are only now being fully realized. Continue reading →

There’s a nice piece in a recent issue of the Economist that captures the excitement and importance of these developments, but also the humbling complexity unfolding in the life sciences around the big questions of genetics. The report riffs on the findings of David Kelley and John Rinn, who study so-called lincRNA, and have discovered, among many other fascinating things, curious relationships between these molecules and transposable elements (my wife is a geneticist whose early interests were in transposons, and she has reminded me that McClintock, in her pioneering maize research, presciently called these “controlling elements“).

Why does this matter to political scientists interested in relationships between our genes and our political attitudes and behaviour?

In one respect it doesn’t: if there are durable heritable roots to the psychological dispositions that inform political phenomena, then we want to be precise about those relationships, not least to know where to direct future research in both genetics and social sciences. We don’t need to worry too much about the causal complexity that leads to those stable heritable dispositions.

On the other hand, science is ultimately about explaining that complexity, so if political scientists want to be players in that game, then simply pointing, as several of these studies do, to correlations and specific clusters of genes, is a good start, but not in itself a very satisfying explanation.

True, many studies of political behaviour proceed as if ideas and environment were everything, and so these ‘genes and politics’ studies are a useful corrective to that view. Still, the humbling possibility is that ongoing work on regulation and expression may force behavioural and population geneticists to rethink some of the durable assumptions behind their methods. While scholars involved in the ‘genes and politics’ field recognize the inordinate complexity of genetic regulation and organism development, they tend to fall back on the familiar old definitions (what a gene is) and assumptions (the stability of protein conformational states), and the methods they’ve inherited from the great innovations of the past in behavioural and population genetics. Those methods remain powerful, but the old assumptions and definitions are, alas, increasingly simplistic or outright mistaken.

Consider this gloss from that Economist article:

Once, and not so long ago, received wisdom was that most of the human genome—perhaps as much as 99% of it—was “junk”. If this junk had a role, it was just to space out the remaining 1%, the genes in which instructions about how to make proteins are encoded, in a useful way in the cell nucleus.

That, it now seems, was about as far from the truth as it is possible to be. The decade or so since the completion of the Human Genome Project has shown that lots of the junk must indeed have a function. The culmination of that demonstration was the publication, in September, of the results of the ENCODE project. This suggested that almost two-thirds of human DNA, rather than just 1% of it, is being copied into molecules of RNA, the chemical that carries protein-making instructions to the sub-cellular factories which turn those proteins out, and that as a consequence, rather than there being just 23,000 genes (namely, the bits of DNA that encode proteins), there may be millions of them.

I think applied behavioural geneticists wandering into the social sciences, and the political scientists adopting their methods, should be humbled by that complexity, even as they do increasingly good work identifying heritable dispositions associated with politics. There is a certain elegance to the statistical and experimental methods these scholars have inherited, but the wonderfully messy complexity of genetic regulation and expression may, as it becomes increasingly well-understood, force us in the social sciences to ask hard questions about just what we are actually explaining with the tools we inherited from an earlier generation.

But of course that’s science.

Polls and Punditry in the U.S. Presidential Race

Posted on by
1

If, like me, you have some geek in you, then you will be heartened by a trend in recent coverage of the U.S. presidential campaign, which pits the statistical prowess of folks like Nate Silver, Sam Wang, Drew Linzer, and Richard Gott and Wesley Colley, on the one hand, against an increasingly defensive tribe of established pundits, on the other (joined recently by the Globe and Mail‘s very own Margaret Wente, who doesn’t let ignorance temper confidence, to judge by this bit of fluff).

Enough has been said about this spat to lead sensible folks to side with the geeks, but a late entry from the pundit camp takes a somewhat novel approach. Michael Gerson of the Washington Post offers some of the usual complaints about the statisticians, and about political science in general (“physics envy!” “subjective values!”), but he also opines thus … Continue reading →

The main problem with this approach to politics is not that it is pseudo-scientific but that it is trivial. An election is not a mathematical equation; it is a nation making a decision. People are weighing the priorities of their society and the quality of their leaders. Those views, at any given moment, can be roughly measured. But spreadsheets don’t add up to a political community. In a democracy, the convictions of the public ultimately depend on persuasion, which resists quantification.

The irony of a former speech-writer for George W. Bush, once labouring in the shadow of Karl Rove, now channelling Rousseau on the general will, should not be lost on the astute reader. As Bush’s campaign strategist, and then White House insider, Rove did far more damage to America’s democracy than any number-crunching academic ever could. Gerson is a part of that sad legacy.

Still, for political theorists, Gerson is singing in a familiar key. Many of us would readily agree that democracy should be more than a simple aggregation of preferences. As citizens we should, ideally, be persuaded by evidence and argument as we reflect on what is in the public interest, and we should vote in light of those judgements.

So, is Gerson offering his remark as an ideal of democracy? Is he suggesting, perhaps, that the number crunchers are pandering to partisan politics as it is, rather than imagining democracy as it might be?

No, he isn’t.

He surely must know that democracy, as we practice it in North America, is ridiculously far from the ideal picture he paints?

If he does, he offers no hint here, least of all when he offers up the following remarkable conceit:

If political punditry has any value in a democracy, it is in clarifying large policy issues and ethical debates …

… wow.

Tell you what, Mike: when you and your fellow partisan pundits — Republican and Democrat — demonstrate a modicum of scientific and philosophical sophistication, then I’ll happily endorse your vision of pundits as public intellectuals, tackling big issues and ethical quandaries, shaping public opinion with reason and evidence, rather than impressions and sound bites.

Until then? I want you and your fellow pundits to stay well away from innocent citizens!

We’ll know tomorrow (hopefully) whether or not (and which of) the number nerds are correct, but if folks like Gerson don’t like the fact that statisticians can now give a (better than) decent guess at election outcomes based on aggregated polling data, then maybe they should take a long hard look at how the United States — and, of course, the rest of us — implement our democratic ideals.

For his part, Gerson could start by apologizing for his involvement with a U.S. administration that never seemed especially friendly to the ideal of democracy as rooted in civil exchanges among informed citizens, seeking the public good together, across the political aisle.

If he did that, then maybe I’d take his high-minded rhetoric seriously.

Assorted Links: Prorogation, Professors, Online Education, and Publishing

Posted on by
1

1. Peter Loewen’s excellent analysis of McGuinty’s decision to prorogue the Ontario Legislature.

And here’s an excellent defence of prorogation.

2. Emmett Macfarlene corrects a number of fundamental myths about what professors do.

3. A new online university education model?

4. “I’m pleased to announce that Mathgen has had its first randomly-generated paper accepted by a reputable journal!”

5. “Just had your paper rejected? Don’t worry — that might boost its ultimate citation tally.”

The Flipped Classroom: Can First Year Students Learn and Apply Rational Choice and Game Theory?

Posted on by
1

As I’ve blogged about here previously, several weeks ago I used the flipped classroom technique to teach my first year seminar students about the “state of nature” and its utility (or lack thereof) for analyzing politics.  The results were exciting and astounding!

The following two weeks, however, were going to be different because I would be introducing my first year students to rational choice and game theory.  In my experience, and even at the graduate level, students tend to have a hard time grasping the basic concepts contained in these theoretical approaches so I figured these weeks would be a good test of the pedagogy.

So what happened? Continue reading →

In the first week, I asked the students to read a chapter from Tom Flanagan’s Game Theory and Canadian Politics, and a short excerpt on social cooperation and the Prisoner’s Dilemma from an introductory textbook. Then they completed an online quiz before class that asked them to: a) provide an example, either from the news or from your own personal experiences, that illustrates and supports any TWO of the following rational choice assumptions: methodological individualism; ordinal vs. cardinal utility; self-interest; and perfect information); b) provide an example, either from the news or from your own personal experiences, that illustrates the prisoner’s dilemma; and c) What did you find most confusing about the readings this week? Using their answers, I wrote two short, mini-lectures on rational choice and game theory, spending most of my time going over the concepts with which the students had trouble.

After delivering the rational choice lecture in class, I divided the students into groups of two or three and asked them to apply the tools of rational choice to explain a number of scenarios.  For instance, they had to craft rational choice explanations of teenage pregnancy, immigration policy; white collar vs. blue collar crime, and an instance of collective action involving community and university actors.  We then took up their answers as a group.

In the first few scenarios, students had trouble developing rational choice explanations for these situations.  What became clear to me very quickly, however, was that they needed some guidance about HOW to apply the theory to the various scenarios.  After some brief explanations about how they might do this, the quality of the group work improved quickly.

After a short break, I began the class with a mini lecture on game theory, again emphasizing the concepts that the students seem to have trouble with in their quiz answers.  After the lecture, I divided the class into groups of three and asked them to solve a particular game theory puzzle.  We plotted their answers on the board and then discussed their answers and what rational choice and game theory would predict.  We then watched the opening bank robbery scene from The Dark Knight, which was similar to the puzzle they had just solved, and I asked the students in their groups to use rational choice and game theory to make sense of what happened to each of the robbers.  We then took up their answers with the larger group.

Finally, we divided up into pairs to play the ultimatum game, first by dividing $10, and then dividing $100.  After plotting their results on the board, I summarized the main experimental findings in the economics literature, before facilitating a discussion about the significance of the results. I then dismissed the class.

A week later, we watched the movie, Return to Paradise, in class.  The movie is in essence, a prisoner’s dilemma.  Midway through the movie, I stopped the film and sketched out, in normal form, the game being played out in the movie.  Before restarting the movie, the class and I worked through the various solutions to the game, varying our assumptions about the players’ preferences and preference structures.  The point here was to show how rational actors should behave, given the scenario presented in the movie. We then watched the rest of the movie.

At the end of the movie, we discussed the following questions: Does rational choice and game theory explain what happened in the movie? Or does this movie provide strong evidence that these theories are not useful when applied to real life situations?

The ensuing discussion was amazing!  The students debated the questions using the language of rational choice and game theory.  They debated whether the rat choice/game theory assumptions about utility maximization, transitive and intransitive preference, and expected vs. actual utility, were successful in accounting for the behaviour of the various characters in the movie. The result was an intelligent and vigorous debate in which students showed that they understood the theories, how they could be applied to various situations, and what some of the strengths and weaknesses were of these approaches.  One student even commented about how rational choice is a post-hoc theory, which is criticism of rational choice and game theory that my students in other classes rarely ever find on their own.

Bottom line: this flipped classroom approach works. Students learn and can apply that learning to various situations and to critical debates.  Next week we tackle, structure and agency, and the role of institutions. Stay tuned.

Diversity and Corporate Canada

Posted on by
Reply

In a recent opinion piece, Ottawa lawyer Catherine McKenna pretty much nails the big problem facing Canadian businesses that hope to compete on the global stage: too often they are led by relics of a bygone (very male, very white, very anglophone) age.

Most Canadian CEOs don’t seem to understand or be able to capitalize on our diverse population. In fact, if you look at the numbers, it’s just the opposite.

The piece is worth a read, and if you’re curious, Catherine is co-president of the Banff Forum, and co-founder of Canadian Lawyers Abroad, both of which try to practice what she preaches in this editorial.

But the editorial isn’t what I want to talk about. No, I want to talk about a particular (and admittedly obscure) response to Catherine’s eminently sensible call for sophisticated and genuine meritocracy in Canadian businesses. Continue reading →

In a series of rambling and bizarrely insulting tweets in reply to Catherine’s piece, an Alberta commentator offers the following remark:

Diversity leads to significantly lower levels of trust in society and business, as per Bowling Alone.

Now, why reply to some obscure tweet that gets Robert Putnam wrong? Because I suspect this popularization of his view is widespread, and it does violence to an important and growing body of research here in Canada and elsewhere, on trust, diversity, immigration, and social welfare policies.

Putnam does find strong evidence that, at the community and neighbourhood level, racial diversity correlates with lower levels of self-reported trust and civic engagement (I add “self-reported” because you really have to trust survey research on public opinion to believe the findings. I do, but others are less enthusiastic).

There has been a lot of fruitful debate around Putnam’s ideas and findings in particular, but that aside, what Putnam pointedly does not find is that diversity undermines trust in structured organizations, like, say, the military and evangelical churches. Or firms.

That’s a different question, and here too there is much fascinating research. The findings to date defy easy summary, but an interested reader could do a lot worse than starting with Scott Page‘s recent book on the subject.

A very rough and sweeping gloss on a complex body of research:

  • several dimensions of diversity can be put to good use within organizations
  • diverse groups are often better at solving problems, but
  • the kinds of problems matter, and the kinds of diversity; indeed,
  • one kind of diversity (race, religion, sex) doesn’t necessarily imply others (problem-solving styles, socioeconomic class, moral beliefs), and unfortunately
  • people generally don’t like arguing across differences of class, or deep moral, political, and ideological differences, so if there are gains to be found from that sort of diversity, the road may be rough, even treacherous (but that doesn’t mean we shouldn’t travel the path, if the benefits outweigh the risks).

But the takeaway lesson here? Catherine pretty much nails it in her editorial, and Robert Putnam’s work doesn’t suggest otherwise.

 

More on Self-Interest, Cooperation, and Game Theory

Posted on by
Reply

Are we fundamentally self-interested or cooperative and altruistic? Recent research suggests:

“It appears from these studies that humans are predisposed to cooperation and generosity, and only become selfish when they take time to think about the situation.”

So what are the implications?

“The particularly intriguing—and somewhat counterintuitive—implications of this body of research are that the traditional ways we try to foster cooperation and generosity may actually be doing the opposite. It is possible when we ask people to consider donating to a cause or to reflect on the benefits of working together, we may actually be promoting greater levels of self-interest.”

From a summary of a new piece in the journal, Nature.

Groups or Individuals? Which are More Likely to Make Decisions in a Game Theoretic Way?

Posted on by
5

I haven’t read the article below yet, but the findings in the abstract remind me of Condorcet’s Jury Theorem, which uses math to show how a group (e.g. a jury) is more likely to reach a correct (and unbiased) decision compared to a single individual (e.g. a judge).

Groups Make Better Self-Interested Decisions

Gary Charness & Matthias Sutter
Journal of Economic Perspectives, Summer 2012, Pages 157–176

Abstract: In this paper, we describe what economists have learned about differences between group and individual decision-making. Continue reading →

This literature is still young, and in this paper, we will mostly draw on experimental work (mainly in the laboratory) that has compared individual decision-making to group decision-making, and to individual decision-making in situations with salient group membership. The bottom line emerging from economic research on group decision-making is that groups are more likely to make choices that follow standard game-theoretic predictions, while individuals are more likely to be influenced by biases, cognitive limitations, and social considerations. In this sense, groups are generally less “behavioral” than individuals. An immediate implication of this result is that individual decisions in isolation cannot necessarily be assumed to be good predictors of the decisions made by groups. More broadly, the evidence casts doubts on traditional approaches that model economic behavior as if individuals were making decisions in isolation.

The Flipped Classroom and University Education: A New Model for Teaching University Classes?

Posted on by
5

This year, I’m teaching a first year seminar (enrollment is 22 first year students) on “Understanding Power and Conflict Through Film: Making Sense of the Politics of the 21st Century.” The idea for this course is to introduce students to some of the main conceptual tools that political scientists use to make sense of the world. My hope is that by the end of the course, the students will be able to apply political science concepts to any political situation they encounter in the news, etc.

Over the summer, I started reading about “the flipped classroom”, a non-traditional way of teaching university courses.  Rather than lecturing at students twice a week, the idea is to design the course so that students do most of the comprehension of readings and themes at home, with class time spent on TEACHING using a variety of techniques. Continue reading →

Using online quizzes before class, the instructor is supposed to identify which parts of the readings or themes that students had trouble with and then use class-time to address these weaknesses.  Some combination of mini-lectures, group work, in-class activities, and discussion, are used to clarify and reinforce the basics. Then the students apply their learning to a variety of real or hypothetical situations/problems, usually in small and larger groups.

Today was the first class.  After going through the usual administrivia about the syllabus, the readings, the website, the assignments, and the like, I gave two mini-lectures, one on how to do critical thinking (with a focus on searching for internal consistency in arguments and testing concepts against a variety of real and hypothetical scenarios) and one on “what is politics” and “what is political science.”  Before class, vial email, I asked the students to read two short readings on these topics but I did not make them do an online quiz since it was the first class.  After completing my two 15 minute lectures on critical thinking and on politics (with a focus on the concepts of idealism, realism, authority, legitimacy, and different types of power and influence), I showed them a movie clip from Golden Balls, a British tv game show that Mike Munger from Duke once sent me.

My students watched the video clip twice: once just to get a sense of what was going on, and once with a set of questions that they had to think about as they watched. The questions asked the students to think about how the clip related to real world political situations, either in the news or in their own life.  It also asked: To what extent does idealism or realism explain what happened in the clip? What does the clip tell us about different types of power (rational persuasion; coercion; and inducements) and the role of authority and legitimacy?

After the second viewing of clip was finished, the students were divided into groups of 2 or 3 and were asked to discuss each of the questions.  After 15 minutes of group discussion, each group presented their answers to the entire class before the class was opened up to general discussion.

The quality of the ensuing discussion was, quite frankly, surprising and amazing!  The students used both of the critical analysis techniques I talked about in my mini lecture (searching for internal consistency and testing against examples); every student participated and every single student correctly applied the concepts of idealism, realism, authority, legitimacy, power, rational persuasion, coercion, and inducements, to analyze the movie cilp and its implications for politics.

I haven’t taught at WLU for very long, but I’ve never seen a group of students so quickly demonstrate comprehension of course concepts, critical thinking and an ability to apply correctly these concepts to a variety of situations. And these are first year students!

Of course, this is just the first class, so I’m sure there will be many bumps along the way. But the results of today’s class were exciting and I look forward to next week’s class.  It has also got me thinking about how I might apply this model to the larger second year courses (enrollment 125) that I usually teach.  The work is substantial, but the payoff is better (and real and more measurable!) learning outcomes (I think!).  Stay tuned!

UPDATE: Additional information about the “flipped classroom” can be found here and a fascinating analysis of online education and educational tools can be found here. Below is a table from the latter link, which is an assessment of various online educational technologies being used these days.

kling_table

ADDITIONAL UPDATE: I’ve been asked to give a brief summary of how “the flipped classroom” works.  Think of the math class you took in high school.  The traditional model is that the teacher introduces a new math concept in class.  Then, students go home to do homework, solving questions relating to the new math concept. They come back to class the next day and the instructor takes up the answers with the entire class. In the flipped classroom, students would go home and watch a video or read a reading on the concept.  They then complete on online quiz on the concept before class.  The instructor reviews the answers from the quiz and in class gives a short lecture on any topics that the students seemed to have trouble with as indicated by their quiz answers.  Then the rest of the class would be in-class homework/application exercises in small groups and large group discussion on the math concept.  Near the end of the class, the instructor assesses how well the students completed the in-class exercises, and ends the class with a short lecture addressing any final weaknesses in comprehension and application.  That’s basically it!

The End of Representative Samples? The Future of Survey Research

Posted on by
Reply

According to Steven Shepard’s article in the National Journal:

“The days of accurate telephone polling are numbered. With more and more Americans dropping their landline service, reliable phone surveys are becoming prohibitively expensive for news organizations and nonprofit groups with tight budgets. Many news outlets are choosing to forgo the rigorous survey research they have commissioned for decades…. With consumer behavior upending traditional polling methods, Zukin of Rutgers predicts that pollsters will stop conducting dual-frame phone surveys (contacting both landline and cell-phone users) “within the next five years. I think we’re going to survey people with whatever mode they wish. That means Internet- and smartphone-based surveys, Zukin says. Indeed, a significant number of the sessions at the pollsters’ conference focused on this kind of research, which uses a methodology known as non-probability, or nonrandom sampling. In many cases, these surveys are completed by respondents who “opt-in,” clicking on a link to complete a poll or joining a Web-based panel (or downloading a smartphone application) to complete surveys—usually with monetary incentives or rewards given for doing so.”

The entire article is worth reading and captures many of the concerns I’ve heard informally from my colleagues who do this type of work.  To me, it sounds like the death knell of survey research, at least in terms of being able to gather data that are representative of the entire population, unless researchers have access to significant financial resources.

Continue reading →

Few academics, however, have access to such resources and so have turned towards web-based panels.  Indeed, my colleague Jason Roy and I used this technique to collect data for our Election Timing paper.

However:

“Some critics in the polling community are highly skeptical of this type of research. Yes, Internet pollsters can create and weight their panels to reflect the public at large, using demographic information to make their samples more representative, they say, but that kind of weighting can serve in some cases to further distort unreliable data.”

I think these criticisms are right.  The only solution is to use web-based panels to target specific publics, rather than representative samples, and to acknowledge upfront the strengths and weaknesses of these techniques.

Weighting is not a solution.  Indeed, I think weighting of any type of dataset, however it is collected, is problematic.  Rarely do you see academic journal articles report findings both with and without weights.  Yet I’ve heard that for some published articles, some of the reported findings that are significant with weights, are either significant at less robust levels or are no longer significant once the weights are removed.

UPDATE: Here’s an article by Andrew Gelman on survey weighting.

Motivation for Youth’s Treatment Scale (MYTS): A New Tool for Measuring Motivation Among Youths and Their Caregivers

Posted on by
Reply

Authors: Carolyn S. Breda and Manuel Riemer
Published March 2012 in Administration and Policy in Mental Health and Mental Health Services Research.

Abstract: Treatment motivation can be important for treatment adherence and outcomes, yet few measures of motivation are available for youths in mental health settings. These authors describe the psychometric properties of the motivation for youth’s treatment scale (MYTS), an 8-item measure with forms for youths and caregivers that assesses their problem recognition and treatment readiness. Results indicate that the MYTS offers practitioners and researchers a brief, psychometrically sound tool for assessing treatment motivation of youths and their caregivers. Multivariate analyses of clinical and non-clinical characteristics of youths and caregivers show that youths’ symptom severity consistently predicts treatment motivation for both groups. However, the strain of caring for the youth adds significantly to caregivers’ recognition of the youth’s troubles. While caregiver and youth motivations correlate, their agreement is low. Caregivers are nearly always more treatment motivated than youths. The authors discuss the implications of their findings for measurement, treatment planning, and future research.

Making Scholarly Data Public?

Posted on by
Reply

Andrew Gelman from the Monkey Cage writes:

“The answer is clear to me: by making your data available, you are making it more likely that others will replicate your results, continue the directions of your research, cite you, etc. Fame and fortune await.”

Yet not all political scientists make their data available:

“If it’s so good to do, why isn’t everybody doing it? Continue reading →

Let’s set aside the cheaters and the insecure people, those scholars who are worried that if someone else gets their hands on their data, they will come to different conclusions. And let’s set aside those researchers who are so clueless that they honestly seem to think that their particular analysis is the last word on the subject ….

I can think of two reasons we (those of us who would actually like our research to be reproduced) don’t routinely share data and code:

1. Effort. This for me is the biggie. As Aven notes, there are social benefits to making data and code available, and as I note above, there are direct personal benefits as well. But these benefits are all medium-to-long-term and they pale beside the short-term costs of getting my act together to put the data in a convenient place. In fact, when I do actually organize my data, it’s often motivated by a desire to make my life easier when handling repeated requests.

2. Rules. The default is for data and code not to be released. Often there are silly IRB rules or commercial restrictions on data. In other cases it seems like too much effort to find out. Again, though, it can be good self-interest to make data available. For example, in our wildly-popular (not yet but eventually, I hope) mrp package in R, we use CCES data, not Annenberg or Pew, for our examples. Why? Because the people at CCES were cool about it. Not only do they release their (old) data for free, they don’t mind us reposting it. Ultimately CCES benefits from this. The freer the data, the easier it will be for people to do analyses, cite CCES, suggest improvements toCCES, etc.”

You can read his full post here:

http://themonkeycage.org/blog/2012/08/03/things-that-arent-prisoners-dilemmas-part-2/