Discuss the criteria for judging the quality of research designs--construct validity, internal validity, external validity, and reliability-- and discuss how you can satisfy these tests with your case study.
Construct validity can be increased by using three tactics. The use of multiple sources of evidence and establishing a chain of evidence are both relevant during the data collection process. The third tactic is to have key informants review the draft case study. The quality of internal validity is challenged when explanatory case studies are used. Difficulties arise during the process to explain how and why one event leads to another, and when making inferences when an event cannot be directly observed. Suggestions for dealing with these challenging include pattern matching, explanation building, addressing rival explanations, and using logic models. External validity is judged by the ability to generalize the case study. Replication logic is used to show external validity when the theory is tested by replicating additional cases. Reliability is judged by the ability of the same case study to be done again and researcher to arrive at the same conclusions. Proper and sufficient documentation is necessary so that the researcher can mimic the same procedures followed the first time. The use of a case study protocol and a case study database is necessary to ensure that there is reliability. Fact: Although multiple-case designs are often more compelling, conducting one may often be too demanding on the time constraints and available resources of a single student. Quote: (37)”…you should be aware of the full range of theories that might be relevant to your study…individual theories-for example, theories of individual development, cognitive behavior, personality, learning and disability, individual perception, and interpersonal interactions;” (There are a couple of theories that could fit my case study from this list- learning and disability and interpersonal interactions) Question: (43) If survey research relies on statistical generalization and case studies rely on analytical generalizations, how do we combine the two if we are utilizing both the case study method and surveys? Would we make more than one (type) of generalization in our study?
Validity of every kind is the issue here! Constructing validity is important during data collection and the use of multiple resources is key. Having a chain of evidence in place and someone to review drafts of your report is helpful. Internal validity takes place during the data analysis stage. The researcher does pattern matching, explanation building, addresses explanations that challenge one another and uses logic models. External validity unlike the others is a barrier in case studies. During a case study and research designing stage the researcher should use theory in single-case studies (some think this is unreliable) and replication logic in multiple-case studies. Reliability is also a very important aspect when collecting data using a protocol is imperative and have a usable database. The chart on page 41 makes knowing these items seem super-duper easy!!! Not a lot of deep thinking here on my part, I just studied the chart…ha!
Question: I am doing pre/post tests and am considering a case study with mixed method of some qualitative data. I worry about this because of all the comments about “one-shot post-test-only” comments…should I go another route?
Quote: Page 30 “If your questions do not lead to the favoring of one unit of analysis over another, your questions are probably either too vague or too numerous---and you may have trouble doing a case study.”
Fact: Reviewing the literature is unavoidable….I have began my search again, and with what I feel is more success.
Yin writes that these four tests have been summarized in numerous textbooks as follows; constuct validity involves identifying correct operational measures for the concepts the researcher is studying, internal validity seeks to establish a causal relationship where specific conditions are believed to lead to others, external validity is defining the area to which the findings of the study can be generalized, and reliability demonstrates that the procedures of a study can not only be replicated but that the same findings will result. For construct validity the researcher needs to be sure to cover two things. First, the researcher needs to define change in terms of specific concepts. Second, the researcher needs to cite published studies that make the same matches. Insuring internal validity involves pattern matching, explanation building, addressing rival explanations and using logic models. Replication logic is key for external validity. For reliability, two specific plans are imperative; the use of a case study protocol and the development of a case study database.
Question: My question is along the same lines as Dana's. Yin writes about the importance of understanding the difference between "analytical generalization" and "statistical generalization." Are analytical generalizations used in case studies rather than statistical generalizations since case studies deal with how and why?
Quote: (p 27) "A research design deals with a logical problem and not a logistical problem."
Fact: (p 53) In some fields such as anthropology and political science, multiple-case studies have been considered a different methodology from single-case studies.
Corey: To what would you attribute the feeling of being more successful this time in your search to review literature?
Ann-- Before I was trying to find technology and a title school or technology and poverty. I focused just on the technology aspect and found some good stuff. I gues I was too specific, but glad I was at first so I know that it doesn't exsist and now I am having to be more broad. How has your luck been with finding literature?
According to Yin, the criteria for judging the quality of research designs include four key tests. These include construct validity, which is defined as identifying and using the correct operational measures for the concepts being studied; internal validity, which refers to seeking to establish a cause and effect relationship in which certain conditions lead to other conditions; external validity, which refers to the ability to to use analytic generalization research findings to generalize to some larger theory beyond one's case study; and the reliability criteria, in which the study yeilds the same results if conducted by others when using the same procedures. To satisify the contstruct validity criteria according to Yin, a case study must use multiple sources of evidence, establish a chain of evidence during the data collection phase of the case study, and have key informants review the final draft in the case study reporting phase. Internal validity can be established during the data analysis phase of the study through pattern matching, explanation building, addressing rival explanations, and using logic models. External validity can be satisfied during the research design phase of research by using theory in single-case studies and by using replication logic in multiple-case studies. To satisfy the reliability criteria for judging the quality of research designs, during the data collection phase of research, researchers should use a case study protocol and develop a case study database.
Quote: (Page 45) "The general way of approaching the reliability problem is to make as many steps as operational as possible and to conduct research as if someone were always looking over your shoulder."
Fact: (Page 32) The desired case should be some real-life phenomenon, not an abstraction such as a topic, an argument, or even a hypothesis.
Question: (Page 34) When designing your case study, Yin emphasizes the importance of identifying and addressing rival explanations for your research findings. How could a reseacher anticipate rival explanations for findings, and then plan accordingly when designing the study?
Corey, in response to your quote about a research question being either too vague or too numerous (page 30), I was a bit relieved to see that Yin says it is okay to "revisit" your unit of analysis as the study develops, and even not to consider "closure" permanent. Yin adds later in the chapter that your case study design may change as new devopments occur and new information surfaces during data collection.
Fact: Yin stresses the importance of sufficient access to data when conducting a case study. Yin even suggests changing your research question if you do not have sufficient access to data. He goes on to reccommend a researcher to choose cases that will shed light on your research questions. This makes since to me.
Quote: Yin describes the research design of an investigation as the "logic that links the data to be collected... to the initial questions of the study." (p. 24)
Question: If I choose not to do a case study for my research, may I just concentrate on the review of literature for my project and forget Yin's book??? ;)
Discussion Question Response: Yin identifies four tests that have been commonly used to establish the quality of any empirical social research: construct validity, internal validity, external validity, and reliability.
Construct validity identifies correct operational measures for the concepts being studied. Yin points out that this test is especially difficult in case study research because critics often highlight how case study investigators fail to develop a sufficiently operational set of measures and use "subjective" judgments to collect data. Yin suggests that during the data collection phase of a case study, the investigator should use multiple sources of evidence and establish a chain of evidence. Yin also suggests to have "key informants" review the draft case study report.
Internal validity for explanatory and causal studies is a test used to establish a causal relationship in which certain conditions are believed to lead to other conditions. A research design fails when the investigator establishes a causal relationship between two factors without knowing that a third factor may have indeed had an effect. Furthermore, since case studies involve the investigator to often make inferences, problems will arise if the inference is incorrect or not all rival explanations considered. Tactics for ensuring internal validity for a research study include pattern matching, explanation building, addressing rival explanations, and using logic models.
External validity deals with the problem of knowing whether a study's findings are able to be generalized. Critics often cite single case studies as a poor basis for generalizing. To overcome this barrier, Yin suggests the use of theory in single-case studies and replication logic in multiple-case studies.
Reliability is a commonly known test that ensures later investigators may follow the same procedures as described by the earlier investigator and conduct the same case study and arrive at the same findings and conclusions. Yin suggests to ensure reliability, an investigator must document all the necessary procedures followed. Moreover, Yin says to make as many steps as operational as possible and to conduct research "as if someone were always looking over your shoulder."
To satisfy construct validity in my study, I plan to use multiple surveys and a test to collect evidence. I also will have a committee of "key informants" that will review my report.
To satisfy internal validity in my study, I plan to explore all rival explanations for my results before establishing any causal relationships between my variables.
To satisfy external validity in my study, I plan to do an in-depth review of literature to analyze theories associated with my topic.
To satisfy reliability in my study, I will have detailed procedures that will be easily repeated by other researchers who will hopefully find the same findings and conclusions as my study.
Barbara, in response to your quote, I don't know about you, but I hate the feeling of someone looking over my shoulder! But, I have to admit that Yin gives some good advice here.
Ann G., in response to your question, I feel that you are spot on in your thinking. Am sure Dr. Hendrix can shed light on this for us.
Corey, in response to your question, I, too, worry about administering a “one-shot post-test-only." My study also involves a few surveys, so hopefully that will save me.
Mrs. Dalrymple, in response your comment regarding the quote I cited about conducting research as if someone is looking over your shoulder, as much as it seems intimidating, it is so true. In research, in the classroom, at the gas station, everywhere-- isn't everything we do monitored by some electronic device? At the risk of sounding paranoid, I think almost everything we do is being recorded and documented to be later used for or against us. In our research, isn't someone (the IRB) going to evaluate every step we take, and judge the validity of our research on our integrity as a researcher? Also, we ultimately are held responsible for our own actions as people, as well as researchers.
Dana, in response to your fact stating multiple case studies are too much for students to handle, I agree! I do think multiple case studies would be the way to go if you were capable (which I am not), because Yin says multiple case designs are usually much stronger than single-case designs. The work would tremendous, but the information gleaned would be rich!
As we all know validity during research is crucial. When judging the quality of the research design we as researchers must consider construct validity, internal validity, external validity and reliability. What I understood about construct validity is that it is using multiple sources of evidence(artifacts) such as pretest, post tests, checkpoints(quizes) and work samples. Of course along with the collection of these items a researcher must keep a spreadsheet on scores and progress, the lack of progress as well as what specific areas are needed for remediation. With this type of validity one would need to involve others such as grade level teachers and principal. Internal validity can take place at the same time as Construct Validity. With internal validity the researcher has to pay attention to the patterns of mastery, non-mastery, give explanation as to why their research participants are not progressing or if they are and why. The researcher must also address why the methods they are using will provide new or improved data which already exists. External validity. With this the researcher must have a well developed theory as to what research they are conducting as well as to have a well defined reason for why they are using a certain research design. How this design fits the research they are conducting. Reliability is the protocol the researcher plans to follow. As a researcher one must have lined out the forms in which are used to conduct research and collect data and specific guides developed on how they will analyze and work with data. The final point of reliability which researchers need to identify, is how they will store the data so that the items and information gathered are not tampered with. A researcher must keep their research under lock and key.
Fact - Page 25 - One pitfall to be avoided, however, is to consider case study designs to be a subset or variant of the research designs used for other methods, such as experiments.
Quote - Page 32 - "To justify using the case study method, you need to go one step further: You need to define as specific, real-life "case" to represent the abstraction."
Question - Page 32 - The concept of "neighboring". So all three 6th grades at my school would be considered a "neighborhood"? Or am I okay to include all three 6th grades in my research? Or should I only use my 6th grade classroom?
Mrs. Dalrymple - I agree with the fact that you shared. If the information being accessed is not adequate or easily accessible it is good to know that it is okay to stop and rethink our research.
Barbara - In response to your fact. I agree! If the research you are doing is not real-life then you are or can be reporting and learning about simply individuals opinions. How would you validate this with data? Sorry I left you with a question.
Dana - In response to your fact. I have learned that this type is challenging. However, this is the design I have chosen. Do you believe I will have success when working with approximately 60 participants?
Construct validity is constructed by three tactics. An investigator may find themselves using multiple sources of evidence, establishing a chain of evidence, and having key informants review draft of the case study report. Internal validity research designs are when the investigator is trying to explain how and why an action led to an additional event. Inferring is also a sign of internal validity. This is where the investigator will infer that results were based on earlier actions. External Validity designs use theory in single-case studies and replication logic in multiple-case studies to further their case study. Reliability tests focus on the use of case study protocol and develop a case study database.
Interesting Fact: There are four different types of theories when developing your case study. These theories are individual theories, group theories, organizational theories, and societal theories. Page 37
Quote: To overcome the barriers to theory development, you should try to prepare for your case study by doing such things as reviewing the literature related to what you would like to study, discussing your topic and ideas with colleagues or teachers, and asking yourself challenging questions about what you are studying, why you are proposing to do the study, and what you hope to learn as a result of the study. Page 37 Question: From Yin’s description it seems that using reliability testing would be more frequently used compared to the other types of testing discussed because it uses other researcher’s procedures. Is this true? Page 45
Question: (43) If survey research relies on statistical generalization and case studies rely on analytical generalizations, how do we combine the two if we are utilizing both the case study method and surveys? Would we make more than one (type) of generalization in our study?
Yes, you would analyze the surveys and then the case(s)separately. You would write up two sections in the end one for each one. Then, you would discuss the findings and results based on what you found from both. I would be careful about generalizations made for both parts though. With small sample sizes in any type of study, you should be hesistant to make a wide-sweeping generalization then. Yes, though to answer your questions, the generalizations look different for surveys than they do for case studies. The intent is different because they answer different types of research questions!
Corey, Question: I am doing pre/post tests and am considering a case study with mixed method of some qualitative data. I worry about this because of all the comments about “one-shot post-test-only” comments…should I go another route?
Not necessarily, you have to know the pros and cons of pretests and posttests. Every method has weaknesses--quantitative and qualitative. A lot of educational research is based on pre and posttest data. Just be careful when you write up your methods and data collection sections and cover validity and reliability there.
Question: (Page 34) When designing your case study, Yin emphasizes the importance of identifying and addressing rival explanations for your research findings. How could a reseacher anticipate rival explanations for findings, and then plan accordingly when designing the study?
This is harder to do if you are filling a hole in the field with your work, but you can look at the key themes from your literature review and see what the scholars said on both sides about their work and those themes that are similar. That will help you anticipate what rivals will say about your own. You can always ask yourself, "What would Creswell say about my research" too. That will help--especially looking back at his evaluation criteria for each type of research method in your fall textbook. That should give you an idea of what other scholars would say about your work. Where would you say your weaknesses are too? You act like an outside reviewer. Play the game. If you are not filling a hole, this is easier because their should be research/reviews out there that point out the pros and cons of it already. Use those when you can.
No, do not skip Yin now--even if you want to use another method. Read the book, do the posts, and the two journals for case studies because a lot of educators do use case studies, and this knowledge will help you read cases better and critically think through them and this method. For the book ad project, you can focus more on your other method though for sure!
Question - Page 32 - The concept of "neighboring". So all three 6th grades at my school would be considered a "neighborhood"? Or am I okay to include all three 6th grades in my research? Or should I only use my 6th grade classroom?
You can do it either way. What are the pros and cons to having all three classes? What are the pros and cons to having one class? Either way, make sure you are really clear in your methods and data collection sections about why you chose to use the class or classes that you did. Another researcher needs to understand that decision and the pros and cons of it.
Question: From Yin’s description it seems that using reliability testing would be more frequently used compared to the other types of testing discussed because it uses other researcher’s procedures. Is this true? Page 45
I would say that reliability and validity are equally significant. You should address both in your methods and data collection sections--how you are making your research reliable and valid for the field.
Tanya, I would also like to know if we could work on our research lit. review instead of a case study. I feel that a case study will not fit with my research.
Ann G- Your quote, "Fact: (p 53) In some fields such as anthropology and political science, multiple-case studies have been considered a different methodology from single-case studies" interests me. I wonder why anthropology and political sciences use multiple-case studies?
Fact: pg 62-..."you should not think that a case study's design cannot be modified by new information or discovery during data collection." What a horrifying thought! To be knee deep in data collection and then have to modify mid stream. Question: In related to the above quote, How does a researcher prevent that? Is there a way? Sometimes I feel like this research is a little like giving birth, you have read all the books, but you never really know what might happen! Quote:pg 53- " The decision to undertake multiple case studies cannot be taken lightly." I fear this quote,because my underlying need to get all the facts/variables accurate makes me think of doing a multiple case study, but my realization of my dis-organization makes me think twice( and this quote!)
Dana, that is a good question about combining about how best to combine analytical and statistical generalizations! That seems to be a hard concept to do, although I see the importance! Corey, I see what you mean about being too specific for lit reviews! I keep trying different words for searching, but to no avail. I will try to be more broad. It seems that I am searching everywhere for the right combination.
The most important quote from this chapter about the testing of validity is "several tactics to be used in dealing with these tests should be applied throughout the subsequent conduct of the case study, not just at the beginning." Construct validity is used when designing the case study and the specific terminology used with what is going to be studied and as you are collecting data. Also, making sure that the operational measures match the concepts( lit review). Internal validity is basically don't infer about a subject or the why the results happened. Commonly when the researcher is explaining the why result. A researcher must first think about these why questions ahead of time to protect internal validity. External validity comes when analyzing the results from the research, when one starts to "decide" since my class showed these results then all classes in my building would act the same way. Replication logic has to be applied if one wants to support behavior in other groups. Another words, to protect external validity, run the test exactly the same. The goal of reliability is to minimize the errors and biases in a study. I realize that this one word, one should keep in mind throughout the process of research design. Along the same way a teacher has to protect the validity of the MAP test in her room. I like the idea of picturing someone looking over my shoulder throughout the data collection and summary.
GREAT POINT! Research is like giving birth. Pilot studies help you work out the tweaks and find any changes that you need to do before the real study. If you have the time, do a pilot study first!
Construct validity can be increased by using three tactics. The use of multiple sources of evidence and establishing a chain of evidence are both relevant during the data collection process. The third tactic is to have key informants review the draft case study. The quality of internal validity is challenged when explanatory case studies are used. Difficulties arise during the process to explain how and why one event leads to another, and when making inferences when an event cannot be directly observed. Suggestions for dealing with these challenging include pattern matching, explanation building, addressing rival explanations, and using logic models. External validity is judged by the ability to generalize the case study. Replication logic is used to show external validity when the theory is tested by replicating additional cases. Reliability is judged by the ability of the same case study to be done again and researcher to arrive at the same conclusions. Proper and sufficient documentation is necessary so that the researcher can mimic the same procedures followed the first time. The use of a case study protocol and a case study database is necessary to ensure that there is reliability.
ReplyDeleteFact: Although multiple-case designs are often more compelling, conducting one may often be too demanding on the time constraints and available resources of a single student.
Quote: (37)”…you should be aware of the full range of theories that might be relevant to your study…individual theories-for example, theories of individual development, cognitive behavior, personality, learning and disability, individual perception, and interpersonal interactions;” (There are a couple of theories that could fit my case study from this list- learning and disability and interpersonal interactions)
Question: (43) If survey research relies on statistical generalization and case studies rely on analytical generalizations, how do we combine the two if we are utilizing both the case study method and surveys? Would we make more than one (type) of generalization in our study?
Validity of every kind is the issue here! Constructing validity is important during data collection and the use of multiple resources is key. Having a chain of evidence in place and someone to review drafts of your report is helpful. Internal validity takes place during the data analysis stage. The researcher does pattern matching, explanation building, addresses explanations that challenge one another and uses logic models. External validity unlike the others is a barrier in case studies. During a case study and research designing stage the researcher should use theory in single-case studies (some think this is unreliable) and replication logic in multiple-case studies. Reliability is also a very important aspect when collecting data using a protocol is imperative and have a usable database. The chart on page 41 makes knowing these items seem super-duper easy!!! Not a lot of deep thinking here on my part, I just studied the chart…ha!
ReplyDeleteQuestion: I am doing pre/post tests and am considering a case study with mixed method of some qualitative data. I worry about this because of all the comments about “one-shot post-test-only” comments…should I go another route?
Quote: Page 30 “If your questions do not lead to the favoring of one unit of analysis over another, your questions are probably either too vague or too numerous---and you may have trouble doing a case study.”
Fact: Reviewing the literature is unavoidable….I have began my search again, and with what I feel is more success.
Yin writes that these four tests have been summarized in numerous textbooks as follows;
ReplyDeleteconstuct validity involves identifying correct operational measures for the concepts the researcher is studying, internal validity seeks to establish a causal relationship where specific conditions are believed to lead to others, external validity is defining the area to which the findings of the study can be generalized, and reliability demonstrates that the procedures of a study can not only be replicated but that the same findings will result.
For construct validity the researcher needs to be sure to cover two things. First, the researcher needs to define change in terms of specific concepts. Second, the researcher needs to cite published studies that make the same matches.
Insuring internal validity involves pattern matching, explanation building, addressing rival explanations and using logic models.
Replication logic is key for external validity.
For reliability, two specific plans are imperative; the use of a case study protocol and the development of a case study database.
Question: My question is along the same lines as Dana's. Yin writes about the importance of understanding the difference between "analytical generalization" and "statistical generalization." Are analytical generalizations used in case studies rather than statistical generalizations since case studies deal with how and why?
Quote: (p 27) "A research design deals with a logical problem and not a logistical problem."
Fact: (p 53) In some fields such as anthropology and political science, multiple-case studies have been considered a different methodology from single-case studies.
Corey: To what would you attribute the feeling of being more successful this time in your search to review literature?
Ann--
ReplyDeleteBefore I was trying to find technology and a title school or technology and poverty. I focused just on the technology aspect and found some good stuff. I gues I was too specific, but glad I was at first so I know that it doesn't exsist and now I am having to be more broad. How has your luck been with finding literature?
According to Yin, the criteria for judging the quality of research designs include four key tests. These include construct validity, which is defined as identifying and using the correct operational measures for the concepts being studied; internal validity, which refers to seeking to establish a cause and effect relationship in which certain conditions lead to other conditions; external validity, which refers to the ability to to use analytic generalization research findings to generalize to some larger theory beyond one's case study; and the reliability criteria, in which the study yeilds the same results if conducted by others when using the same procedures. To satisify the contstruct validity criteria according to Yin, a case study must use multiple sources of evidence, establish a chain of evidence during the data collection phase of the case study, and have key informants review the final draft in the case study reporting phase. Internal validity can be established during the data analysis phase of the study through pattern matching, explanation building, addressing rival explanations, and using logic models. External validity can be satisfied during the research design phase of research by using theory in single-case studies and by using replication logic in multiple-case studies. To satisfy the reliability criteria for judging the quality of research designs, during the data collection phase of research, researchers should use a case study protocol and develop a case study database.
ReplyDeleteQuote: (Page 45) "The general way of approaching the reliability problem is to make as many steps as operational as possible and to conduct research as if someone were always looking over your shoulder."
Fact: (Page 32) The desired case should be some real-life phenomenon, not an abstraction such as a topic, an argument, or even a hypothesis.
Question: (Page 34) When designing your case study, Yin emphasizes the importance of identifying and addressing rival explanations for your research findings. How could a reseacher anticipate rival explanations for findings, and then plan accordingly when designing the study?
Corey, in response to your quote about a research question being either too vague or too numerous (page 30), I was a bit relieved to see that Yin says it is okay to "revisit" your unit of analysis as the study develops, and even not to consider "closure" permanent. Yin adds later in the chapter that your case study design may change as new devopments occur and new information surfaces during data collection.
ReplyDeleteFact: Yin stresses the importance of sufficient access to data when conducting a case study. Yin even suggests changing your research question if you do not have sufficient access to data. He goes on to reccommend a researcher to choose cases that will shed light on your research questions. This makes since to me.
ReplyDeleteQuote: Yin describes the research design of an investigation as the "logic that links the data to be collected... to the initial questions of the study." (p. 24)
Question: If I choose not to do a case study for my research, may I just concentrate on the review of literature for my project and forget Yin's book??? ;)
Discussion Question Response:
ReplyDeleteYin identifies four tests that have been commonly used to establish the quality of any empirical social research: construct validity, internal validity, external validity, and reliability.
Construct validity identifies correct operational measures for the concepts being studied. Yin points out that this test is especially difficult in case study research because critics often highlight how case study investigators fail to develop a sufficiently operational set of measures and use "subjective" judgments to collect data. Yin suggests that during the data collection phase of a case study, the investigator should use multiple sources of evidence and establish a chain of evidence. Yin also suggests to have "key informants" review the draft case study report.
Internal validity for explanatory and causal studies is a test used to establish a causal relationship in which certain conditions are believed to lead to other conditions. A research design fails when the investigator establishes a causal relationship between two factors without knowing that a third factor may have indeed had an effect. Furthermore, since case studies involve the investigator to often make inferences, problems will arise if the inference is incorrect or not all rival explanations considered. Tactics for ensuring internal validity for a research study include pattern matching, explanation building, addressing rival explanations, and using logic models.
External validity deals with the problem of knowing whether a study's findings are able to be generalized. Critics often cite single case studies as a poor basis for generalizing. To overcome this barrier, Yin suggests the use of theory in single-case studies and replication logic in multiple-case studies.
Reliability is a commonly known test that ensures later investigators may follow the same procedures as described by the earlier investigator and conduct the same case study and arrive at the same findings and conclusions. Yin suggests to ensure reliability, an investigator must document all the necessary procedures followed. Moreover, Yin says to make as many steps as operational as possible and to conduct research "as if someone were always looking over your shoulder."
To satisfy construct validity in my study, I plan to use multiple surveys and a test to collect evidence. I also will have a committee of "key informants" that will review my report.
To satisfy internal validity in my study, I plan to explore all rival explanations for my results before establishing any causal relationships between my variables.
To satisfy external validity in my study, I plan to do an in-depth review of literature to analyze theories associated with my topic.
To satisfy reliability in my study, I will have detailed procedures that will be easily repeated by other researchers who will hopefully find the same findings and conclusions as my study.
Barbara, in response to your quote, I don't know about you, but I hate the feeling of someone looking over my shoulder! But, I have to admit that Yin gives some good advice here.
ReplyDeleteAnn G., in response to your question, I feel that you are spot on in your thinking. Am sure Dr. Hendrix can shed light on this for us.
Corey, in response to your question, I, too, worry about administering a “one-shot post-test-only." My study also involves a few surveys, so hopefully that will save me.
Mrs. Dalrymple, in response your comment regarding the quote I cited about conducting research as if someone is looking over your shoulder, as much as it seems intimidating, it is so true. In research, in the classroom, at the gas station, everywhere-- isn't everything we do monitored by some electronic device? At the risk of sounding paranoid, I think almost everything we do is being recorded and documented to be later used for or against us. In our research, isn't someone (the IRB) going to evaluate every step we take, and judge the validity of our research on our integrity as a researcher? Also, we ultimately are held responsible for our own actions as people, as well as researchers.
ReplyDeleteDana, in response to your fact stating multiple case studies are too much for students to handle, I agree! I do think multiple case studies would be the way to go if you were capable (which I am not), because Yin says multiple case designs are usually much stronger than single-case designs. The work would tremendous, but the information gleaned would be rich!
ReplyDeleteAs we all know validity during research is crucial. When judging the quality of the research design we as researchers must consider construct validity, internal validity, external validity and reliability. What I understood about construct validity is that it is using multiple sources of evidence(artifacts) such as pretest, post tests, checkpoints(quizes) and work samples. Of course along with the collection of these items a researcher must keep a spreadsheet on scores and progress, the lack of progress as well as what specific areas are needed for remediation. With this type of validity one would need to involve others such as grade level teachers and principal. Internal validity can take place at the same time as Construct Validity. With internal validity the researcher has to pay attention to the patterns of mastery, non-mastery, give explanation as to why their research participants are not progressing or if they are and why. The researcher must also address why the methods they are using will provide new or improved data which already exists.
ReplyDeleteExternal validity. With this the researcher must have a well developed theory as to what research they are conducting as well as to have a well defined reason for why they are using a certain research design. How this design fits the research they are conducting.
Reliability is the protocol the researcher plans to follow. As a researcher one must have lined out the forms in which are used to conduct research and collect data and specific guides developed on how they will analyze and work with data. The final point of reliability which researchers need to identify, is how they will store the data so that the items and information gathered are not tampered with. A researcher must keep their research under lock and key.
Fact - Page 25 - One pitfall to be avoided, however, is to consider case study designs to be a subset or variant of the research designs used for other methods, such as experiments.
ReplyDeleteQuote - Page 32 - "To justify using the case study method, you need to go one step further: You need to define as specific, real-life "case" to represent the abstraction."
Question - Page 32 - The concept of "neighboring". So all three 6th grades at my school would be considered a "neighborhood"? Or am I okay to include all three 6th grades in my research? Or should I only use my 6th grade classroom?
Mrs. Dalrymple - I agree with the fact that you shared. If the information being accessed is not adequate or easily accessible it is good to know that it is okay to stop and rethink our research.
ReplyDeleteBarbara - In response to your fact. I agree! If the research you are doing is not real-life then you are or can be reporting and learning about simply individuals opinions. How would you validate this with data? Sorry I left you with a question.
Dana - In response to your fact. I have learned that this type is challenging. However, this is the design I have chosen. Do you believe I will have success when working with approximately 60 participants?
ReplyDeleteConstruct validity is constructed by three tactics. An investigator may find themselves using multiple sources of evidence, establishing a chain of evidence, and having key informants review draft of the case study report. Internal validity research designs are when the investigator is trying to explain how and why an action led to an additional event. Inferring is also a sign of internal validity. This is where the investigator will infer that results were based on earlier actions. External Validity designs use theory in single-case studies and replication logic in multiple-case studies to further their case study. Reliability tests focus on the use of case study protocol and develop a case study database.
ReplyDeleteInteresting Fact: There are four different types of theories when developing your case study. These theories are individual theories, group theories, organizational theories, and societal theories. Page 37
Quote: To overcome the barriers to theory development, you should try to prepare for your case study by doing such things as reviewing the literature related to what you would like to study, discussing your topic and ideas with colleagues or teachers, and asking yourself challenging questions about what you are studying, why you are proposing to do the study, and what you hope to learn as a result of the study. Page 37
Question: From Yin’s description it seems that using reliability testing would be more frequently used compared to the other types of testing discussed because it uses other researcher’s procedures. Is this true? Page 45
Dana,
ReplyDeleteQuestion: (43) If survey research relies on statistical generalization and case studies rely on analytical generalizations, how do we combine the two if we are utilizing both the case study method and surveys? Would we make more than one (type) of generalization in our study?
Yes, you would analyze the surveys and then the case(s)separately. You would write up two sections in the end one for each one. Then, you would discuss the findings and results based on what you found from both. I would be careful about generalizations made for both parts though. With small sample sizes in any type of study, you should be hesistant to make a wide-sweeping generalization then. Yes, though to answer your questions, the generalizations look different for surveys than they do for case studies. The intent is different because they answer different types of research questions!
Great job here!!!! Keep it up!!!!
Corey,
ReplyDeleteQuestion: I am doing pre/post tests and am considering a case study with mixed method of some qualitative data. I worry about this because of all the comments about “one-shot post-test-only” comments…should I go another route?
Not necessarily, you have to know the pros and cons of pretests and posttests. Every method has weaknesses--quantitative and qualitative. A lot of educational research is based on pre and posttest data. Just be careful when you write up your methods and data collection sections and cover validity and reliability there.
Great job here!!!! Keep it up!!!
Barb,
ReplyDeleteQuestion: (Page 34) When designing your case study, Yin emphasizes the importance of identifying and addressing rival explanations for your research findings. How could a reseacher anticipate rival explanations for findings, and then plan accordingly when designing the study?
This is harder to do if you are filling a hole in the field with your work, but you can look at the key themes from your literature review and see what the scholars said on both sides about their work and those themes that are similar. That will help you anticipate what rivals will say about your own. You can always ask yourself, "What would Creswell say about my research" too. That will help--especially looking back at his evaluation criteria for each type of research method in your fall textbook. That should give you an idea of what other scholars would say about your work. Where would you say your weaknesses are too? You act like an outside reviewer. Play the game. If you are not filling a hole, this is easier because their should be research/reviews out there that point out the pros and cons of it already. Use those when you can.
Great job here!!!!! Keep it up!!!!
Tanya,
ReplyDeleteNo, do not skip Yin now--even if you want to use another method. Read the book, do the posts, and the two journals for case studies because a lot of educators do use case studies, and this knowledge will help you read cases better and critically think through them and this method. For the book ad project, you can focus more on your other method though for sure!
Great job here!!! Keep it up!!!!!
Tanya,
ReplyDeleteGood thoughts here on your validity and reliability issues for your own study! You are heading in the right direction!
Ann,
ReplyDeleteQuestion - Page 32 - The concept of "neighboring". So all three 6th grades at my school would be considered a "neighborhood"? Or am I okay to include all three 6th grades in my research? Or should I only use my 6th grade classroom?
You can do it either way. What are the pros and cons to having all three classes? What are the pros and cons to having one class? Either way, make sure you are really clear in your methods and data collection sections about why you chose to use the class or classes that you did. Another researcher needs to understand that decision and the pros and cons of it.
Great job here!!!! Keep up the good work!!!
Chris,
ReplyDeleteQuestion: From Yin’s description it seems that using reliability testing would be more frequently used compared to the other types of testing discussed because it uses other researcher’s procedures. Is this true? Page 45
I would say that reliability and validity are equally significant. You should address both in your methods and data collection sections--how you are making your research reliable and valid for the field.
Great job here!!! Keep up the good work!!!
Tanya, I would also like to know if we could work on our research lit. review instead of a case study. I feel that a case study will not fit with my research.
ReplyDeleteAnn G- Your quote, "Fact: (p 53) In some fields such as anthropology and political science, multiple-case studies have been considered a different methodology from single-case studies" interests me. I wonder why anthropology and political sciences use multiple-case studies?
Fact: pg 62-..."you should not think that a case study's design cannot be modified by new information or discovery during data collection."
ReplyDeleteWhat a horrifying thought! To be knee deep in data collection and then have to modify mid stream.
Question: In related to the above quote, How does a researcher prevent that? Is there a way? Sometimes I feel like this research is a little like giving birth, you have read all the books, but you never really know what might happen!
Quote:pg 53- " The decision to undertake multiple case studies cannot be taken lightly." I fear this quote,because my underlying need to get all the facts/variables accurate makes me think of doing a multiple case study, but my realization of my dis-organization makes me think twice( and this quote!)
Dana, that is a good question about combining about how best to combine analytical and statistical generalizations! That seems to be a hard concept to do, although I see the importance!
Corey, I see what you mean about being too specific for lit reviews! I keep trying different words for searching, but to no avail. I will try to be more broad. It seems that I am searching everywhere for the right combination.
The most important quote from this chapter about the testing of validity is "several tactics to be used in dealing with these tests should be applied throughout the subsequent conduct of the case study, not just at the beginning."
ReplyDeleteConstruct validity is used when designing the case study and the specific terminology used with what is going to be studied and as you are collecting data. Also, making sure that the operational measures match the concepts( lit review).
Internal validity is basically don't infer about a subject or the why the results happened. Commonly when the researcher is explaining the why result. A researcher must first think about these why questions ahead of time to protect internal validity.
External validity comes when analyzing the results from the research, when one starts to "decide" since my class showed these results then all classes in my building would act the same way. Replication logic has to be applied if one wants to support behavior in other groups. Another words, to protect external validity, run the test exactly the same.
The goal of reliability is to minimize the errors and biases in a study. I realize that this one word, one should keep in mind throughout the process of research design. Along the same way a teacher has to protect the validity of the MAP test in her room. I like the idea of picturing someone looking over my shoulder throughout the data collection and summary.
Shelly,
ReplyDeleteGREAT POINT! Research is like giving birth. Pilot studies help you work out the tweaks and find any changes that you need to do before the real study. If you have the time, do a pilot study first!
Keep up the good work!!!!