When a researcher reports that a result is statistically significant What does it mean?

Asked by Tdyates921

When a researcher reports that a result is statistically significant, what does this mean?

Group of answer choices

The result is important and clinically meaningful

The result does not contain a Type I or II error

The obtained result likely was not the result of chance

The result did not fall within the critical region on the sampling distribution

Answer & Explanation

Solved by verified expert

Answered by tutorhaniel

risus ante, dapibus a molestie consequat, ultrices ac m

gue vel laoreet ac, dictum vitae odio. Donec aliquet. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam lacinia pulvinar tortor nec facilisis. Pellentesque dapibus efficitur laoreet. Nam risus ante, dapibus a molestie consequat, ultrices ac magna. Fusce dui

Unlock full access to Course Hero

Explore over 16 million step-by-step answers from our library

Subscribe to view answer

Step-by-step explanation

acinia pulvinar tortor nec facilisis. Pellentesque dapibus efficitur laoreet. Nam risus ante, dapibus a molestie consequat, ultrices ac magna. Fusce d

molestie consequat, ultrices ac magna. Fusce dui lectus, congue vel laoreet ac, dictum vitae odio. Donec aliquet. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Na

Student reviews

100% [3 ratings]

PPA 696 RESEARCH METHODS

TESTS FOR SIGNIFICANCE

What are Tests for Significance
Steps in Testing for Statistical Significance
1] State the Research Hypothesis
2] State the Null Hypothesis
3] Type I and Type II Errors
Select a probability of error level [alpha level]
4] Chi Square Test
Calculate Chi Square
Degrees of freedom
Distribution Tables
Interpret the results
5] T-Test
Calculate T-Test
Degrees of freedom
Distribution Tables
Interpret the results
Reporting Tests of Statistical Significance
Final Comments

What Are Tests for Significance

    Two questions arise about any hypothesized relationship between two variables: 1] what is the probability that the relationship exists; 2] if it does, how strong is the relationship     There are two types of tools that are used to address these questions: the first is addressed by tests for statistical significance; and the second is addressed by Measures of Association.

    Tests for statistical significance are used to address the question: what is the probability that what we think is a relationship between two variables is really just a chance occurrence?

    If we selected many samples from the same population, would we still find the same relationship between these two variables in every sample? If we could do a census of the population, would we also find that this relationship exists in the population from which the sample was drawn? Or is our finding due only to random chance?

    Tests for statistical significance tell us what the probability is that the relationship we think we have found is due only to random chance. They tell us what the probability is that we would be making an error if we assume that we have found that a relationship exists.

    We can never be completely 100% certain that a relationship exists between two variables. There are too many sources of error to be controlled, for example, sampling error, researcher bias, problems with reliability and validity, simple mistakes, etc.

    But using probability theory and the normal curve, we can estimate the probability of being wrong, if we assume that our finding a relationship is true. If the probability of being wrong is small, then we say that our observation of the relationship is a statistically significant finding.

    Statistical significance means that there is a good chance that we are right in finding that a relationship exists between two variables. But statistical significance is not the same as practical significance. We can have a statistically significant finding, but the implications of that finding may have no practical application. The researcher must always examine both the statistical and the practical significance of any research finding.

    For example, we may find that there is a statistically significant relationship between a citizen's age and satisfaction with city recreation services. It may be that older citizens are 5% less satisfied than younger citizens with city recreation services. But is 5% a large enough difference to be concerned about?

    Often times, when differences are small but statistically significant, it is due to a very large sample size; in a sample of a smaller size, the differences would not be enough to be statistically significant.
 

Steps in Testing for Statistical Significance

1] State the Research Hypothesis
2] State the Null Hypothesis
3] Select a probability of error level [alpha level]
4] Select and compute the test for statistical significance
5] Interpret the results
 

1] State the Research Hypothesis

    A research hypothesis states the expected relationship between two variables. It may be stated in general terms, or it may include dimensions of direction and magnitude. For example, General: The length of the job training program is related to the rate of job placement of trainees. Direction: The longer the training program, the higher the rate of job placement of trainees. Magnitude: Longer training programs will place twice as many trainees into jobs as shorter programs. General: Graduate Assistant pay is influenced by gender. Direction: Male graduate assistants are paid more than female graduate assistants. Magnitude: Female graduate assistants are paid less than 75% of what male graduate assistants are paid.

2] State the Null Hypothesis

    A null hypothesis usually states that there is no relationship between the two variables. For example, There is no relationship between the length of the job training program and the rate of job placement of trainees. Graduate assistant pay is not influenced by gender.     A null hypothesis may also state that the relationship proposed in the research hypothesis is not true. For example, Longer training programs will place the same number or fewer trainees into jobs as shorter programs. Female graduate assistants are paid at least 75% or more of what male graduate assistants are paid.     Researchers use a null hypothesis in research because it is easier to disprove a null hypothesis than it is to prove a research hypothesis. The null hypothesis is the researcher's "straw man." That is, it is easier to show that something is false once than to show that something is always true. It is easier to find disconfirming evidence against the null hypothesis than to find confirming evidence for the research hypothesis.
 

3] TYPE I AND TYPE II ERRORS

    Even in the best research project, there is always a possibility [hopefully a small one] that the researcher will make a mistake regarding the relationship between the two variables. There are two possible mistakes or errors.

    The first is called a Type I error. This occurs when the researcher assumes that a relationship exists when in fact the evidence is that it does not. In a Type I error, the researcher should accept the null hypothesis and reject the research hypothesis, but the opposite occurs. The probability of committing a Type I error is called alpha.

    The second is called a Type II error. This occurs when the researcher assumes that a relationship does not exist when in fact the evidence is that it does. In a Type II error, the researcher should reject the null hypothesis and accept the research hypothesis, but the opposite occurs. The probability of committing a Type II error is called beta.

    Generally, reducing the possibility of committing a Type I error increases the possibility of committing a Type II error and vice versa, reducing the possibility of committing a Type II error increases the possibility of committing a Type I error.

    Researchers generally try to minimize Type I errors, because when a researcher assumes a relationship exists when one really does not, things may be worse off than before. In Type II errors, the researcher misses an opportunity to confirm that a relationship exists, but is no worse off than before.

In this example, which type of error would you prefer to commit? Research Hypothesis: El Nino has reduced crop yields in County X, making it eligible for government disaster relief. Null Hypothesis: El Nino has not reduced crop yields in County X, making it ineligible for government disaster relief.     If a Type I error is committed, then the County is assumed to be eligible for disaster relief, when it really is not [the null hypothesis should be accepted, but it is rejected]. The government may be spending disaster relief funds when it should not, and taxes may be raised.

    If a Type II error is committed, then the County is assumed to be ineligible for disaster relief, when it really is eligible [the null hypothesis should be accepted, but it is rejected]. The government may not spend disaster relief funds when it should, and farmers may go into bankruptcy.

In this example, which type of error would you prefer to commit? Research Hypothesis: The new drug is better at treating heart attacks than the old drug Null Hypothesis: The new drug is no better at treating heart attacks than the old drug     If a Type I error is committed, then the new drug is assumed to be better when it really is not [the null hypothesis should be accepted, but it is rejected]. People may be treated with the new drug, when they would have been better off with the old one.

    If a Type II error is committed, then the new drug is assumed to be no better when it really is better [the null hypothesis should be rejected, but it is accepted]. People may not be treated with the new drug, although they would be better off than with the old one.
 

SELECT A PROBABILITY OF ERROR LEVEL [ALPHA LEVEL]

    Researchers generally specify the probability of committing a Type I error that they are willing to accept, i.e., the value of alpha. In the social sciences, most researchers select an alpha=.05. This means that they are willing to accept a probability of 5% of making a Type I error, of assuming a relationship between two variables exists when it really does not. In research involving public health, however, an alpha of .01 is not unusual. Researchers do not want to have a probability of being wrong more than 0.1% of the time, or one time in a thousand.

    If the relationship between the two variables is strong [as assessed by a Measure of Association], and the level chosen for alpha is .05, then moderate or small sample sizes will detect it. As relationships get weaker, however, and/or as the level of alpha gets smaller, larger sample sizes will be needed for the research to reach statistical significance.
 

4] The Chi Square Test

    For nominal and ordinal data, Chi Square is used as a test for statistical significance. For example, we hypothesize that there is a relationship between the type of training program attended and the job placement success of trainees. We gather the following data:

Type of Training Attended:	Number attending Training
Vocational Education	200
Work Skills Training	250
Total	450

 
 

Placed in a Job?	Number of Trainees
Yes	300
No	150
Total	450

    To compute Chi Square, a table showing the joint distribution of the two variables is needed:

Table 1. Job Placement by Type of Training [Observed Frequencies]
 

Placed in a Job?	Type of Training
Vocational Education	Work Skills Training	Total
Yes	175	125	300
No	25	125	150
Total	200	250	450

    Chi Square is computed by looking at the different parts of the table. The "cells" of the table are the squares in the middle of the table containing numbers that are completely enclosed. The cells contain the frequencies that occur in the joint distribution of the two variables. The frequencies that we actually find in the data are called the "observed" frequencies.

    In this table, the cells contain the frequencies for vocational education trainees who got a job [n=175] and who didn't get a job [n=25], and the frequencies for work skills trainees who got a job [n=125] and who didn't get a job [n=125].

    The "Total" columns and rows of the table show the marginal frequencies. The marginal frequencies are the frequencies that we would find if we looked at each variable separately by itself. For example, we can see in the "Total" column that there were 300 people who got a job and 150 people who didn't. We can see in the "Total" row that there were 200 people in vocational education training and 250 people in job skills training.

    Finally, there is the total number of observations in the whole table, called N. In this table, N=450.
 

Calculate Chi Square

1] display observed frequencies for each cell
2] calculate expected frequencies for each cell
3] calculate, for each cell, the expected minus observed frequency squared, divided by the expected frequency
4] all up the results for all the cells

    To find the value of Chi Square, we first assume that there is no relationship between the type of training program attended and whether the trainee was placed in a job. If we look at the column total, we can see that 300 of 450 people found a job, or 66.7% of the total people in training found a job. We can also see that 150 of 450 people did not find a job, or 33.3% of the total people in training did not find a job.

    If there was no relationship between the type of program attended and success in finding a job, then we would expect 66.7% of trainees of both types of training programs to get a job, and 33.3% of both types of training programs to not get a job.

    The first thing that Chi Square does is to calculate "expected" frequencies for each cell. The expected frequency is the frequency that we would have expected to appear in each cell if there was no relationship between type of training program and job placement.

    The way to calculate the expected cell frequency is to multiply the column total for that cell, by the row total for that cell, and divide by the total number of observations for the whole table.

For the upper left hand corner cell, multiply 200 by 300 and divide by 450=133.3
For the lower left hand corner cell, multiply 200 by 150 and divide by 450=66.7
For the upper right hand corner cell, multiply 250 by 300 and divide by 450=166.7
For the lower right hand corner cell, multiply 250 by 150 and divide by 450=83.3

Table 2. Job Placement by Type of Training [Expected Frequencies]
 

Placed in a Job?	Type of Training
Vocational Education	Work Skills Training	Total
Yes	133.3	166.7	300
No	66.7	83.3	150
Total	200	250	450

    This table shows the distribution of "expected" frequencies, that is, the cell frequencies we would expect to find if there was no relationship between type of training and job placement.

    Note that Chi Square is not reliable if any cell in the contingency table has an expected frequency of less than 5.

    To calculate Chi Square, we need to compare the original, observed frequencies with the new, expected frequencies. For each cell, we perform the following calculations:
a] Subtract the value of the observed frequency from the value of the expected frequency
b] square the result
c] divide the result by the value of the expected frequency

For each cell above,
 

fe - fo	[fe - fo]2	[[fe - fo]2] / fe	Result
[133.3 - 175]	[133.3 - 175]2	[[133.3 - 175]2] / 133.3	13.04
[66.7 - 25]	[66.7 - 25]2	[[66.7 - 25]2] / 66.7	26.07
[166.7 - 125]	[166.7 - 125]2	[[166.7 - 125]2] / 166.7	10.43
[83.3 - 125]	[83.3 - 125]2	[[83.3 - 135]2] / 83.3	20.88

To calculate the value of Chi Square, add up the results for each cell--Total=70.42
 

DEGREES OF FREEDOM

    We cannot interpret the value of the Chi Square statistics by itself. Instead, we must put it into a context.

    In theory, the value of the Chi Square statistic is normally distributed; that is, the value of the Chi Square statistics looks like a normal [bell-shaped] curve. Thus we can use the properties of the normal curve to interpret the value obtained from our calculation of the Chi Square statistic.

    If the value we obtain for Chi Square is large enough, then we can say that it indicates the level of statistical significance at which the relationship between the two variables can be presumed to exist.

    However, whether the value is large enough depends on two things: the size of the contingency table from which the Chi Square statistic has been computed; and the level of alpha that we have selected.

    The larger the size of the contingency table, the larger the value of Chi Square will need to be in order to reach statistical significance, if other things are equal. Similarly, the more stringent the level of alpha, the larger the value of Chi Square will need to be, in order to reach statistical significance, if other things are equal.

    The term "degrees of freedom" is used to refer to the size of the contingency table on which the value of the Chi Square statistic has been computed. The degrees of freedom is calculated as the product of [the number of rows in the table minus 1] times [the number of columns in the table minus ].
 

For a table with two rows of cells and two columns of cells, the formula is: df = [2 - 1] x [2 - 1] = [1] x [1] = 1 For a table with two rows of cells and three columns of cells, the formula is: df = [3 - 1] x [2 - 1] = [2] x [1] = 2 For a table with three rows of cells and three columns of cells, the formula is: df = [3 - 1] x [3 - 1] = [2] x [2] = 4     The level of alpha can vary, but the smaller the value, the more stringent the requirement for reaching statistical significance becomes. Alpha levels are often written as the "p-value", or "p=.05." Usual levels are p=.05 [or the chance of one in 20 of making an error], or p=.01 [or the chance of one in 100 of making an error], or p=.001 [or the chance of one in 1,000 of making an error].

    When reporting the level of alpha, it is usually reported as being "less than" some level, using the "less than" sign or