How are Programs on the CEBC Reviewed?
As a new or experienced user of the CEBC website, you might wonder how the CEBC review process works; including how the level of Child Welfare System Relevance is determined, and whether programs can be rated on the Scientific Rating Scale.
How are Topic Areas Chosen?
Numerous topic areas are relevant to child welfare; therefore, the CEBC Advisory Committee is consulted annually to determine the topic areas that will be added to the website during the grant’s next fiscal year. Once those topic areas are chosen, a topic expert for each area is recruited with the consultation of the CEBC Scientific Panel. CEBC staff works closely with each topic expert to draft a clear definition of the topic area.
How are Programs within the Topic Areas Chosen?
Once the definition of the topic area is finalized, the topic expert generates a list of potential programs for the topic area. CEBC staff also conducts an extensive literature search at the same time to see what programs could be in the topic area. The lists are then combined and reviewed with the topic expert and a final list is created. All programs listed on the CEBC website are included in one or more of the topic areas highlighted on the website.
How is the Information about the Program Obtained?
Once the list is finalized, screening questions are sent to each program to ensure the program has a manual or offers training, to establish the legitimacy of the contact’s relationship to the program, to collect the program’s goals, and to obtain the contact’s willingness to fill out a detailed questionnaire about the program. After the program passes the screening process, a questionnaire is sent out to the program representative to be filled out in a timely manner. Once the questionnaire is received back, the research review process begins. A member of the CEBC staff also conducts a literature search on each program to obtain any published, peer reviewed research. If a program does not return the questionnaire or refuses to fill it out, then it is considered a nonresponder. If sufficient information is publicly available on the program an abbreviated description of the program is created by CEBC staff and posted on the website.
How is the Program Information Processed?
For the programs where the program representative completed the questionnaire, CEBC staff reviews and edits what is now considered the program outline. The program representative is contacted for any needed clarification.Published, peer-reviewed articles (i.e., research evidence) that report the outcomes of a research study are summarized and added to the program outline. For the programs from nonresponders, any published, peer-reviewed articles that report the outcomes of a research study are also summarized. The summaries are then edited and added to the nonresponder program outline.
What Determines Whether a Program Can Be Rated on the CEBC Scientific Rating Scale?
All outcomes from research studies on a program must have been reported in a published, peer-reviewed journal to be considered by the CEBC staff during the rating process. Please see the Scientific Rating Scale for all of the criteria needed to be rated in one of the five categories on the scale. If a program does not fit the criteria for any of the categories on the scale, it is determined to be NR (Not able to be Rated).
How is an Eligible Program Rated and Evaluated?
Outlines for programs that are able to be rated are sent to the raters along with the published, peer-reviewed research articles and with a copy of the Scientific Rating Scale. Typically, the raters are the topic expert, and two of the CEBC staff. If there is a discrepancy, the CEBC Scientific Director makes the final decision. In addition to a rating on the Scientific Rating Scale, each eligible program is examined to see which child welfare outcome(s) (e.g., safety, permanency, and/or child/family well-being) are addressed in the research evidence for that program.
How is the Child Welfare System Relevance Level of a Program Determined?
When is the Program Loaded onto the CEBC Website?
The edited program outline is then finalized by adding the Scientific Rating and Child Welfare Outcomes information, if applicable; the Child Welfare System Relevance Level; and any standardized language used by the website. Then it is sent, with a passive agreement deadline, to the program representative (even if he/she is a nonresponder) for one last review before the program is loaded onto the website. Once a majority of the programs for a new topic area are ready, then the topic area and new programs are added to the CEBC website and announced in a CEBC email alert.
How is a Program Added to Existing Topic Areas?
A program that is brought to the attention of the CEBC either by its users or by itself is taken into consideration by the CEBC staff. To be eligible to be added to the CEBC, the program must fit into one of the existing topic areas on the website and have either enough research evidence to obtain a rating on the Scientific Rating Scale or be widely used and/or marketed in California. Once it is deemed eligible, the review process is very similar to the one listed above except that the topic expert is usually not one of the raters, instead an additional rater from the CEBC staff is used. The other notable difference is that the program is loaded onto the website when there are enough other programs or new information to warrant sending out a CEBC email alert.
How is a Program Re-Rated?
Once the CEBC staff becomes aware of new research evidence on the program, either through the program’s representative or the bi-annual literature review process, the program re-starts the CEBC rating process described above. However, the raters do not usually include the topic expert, but instead an additional rater from the CEBC staff. Once the rating process is finished, the program representative is alerted about the rating change. After that, the program information is updated on the CEBC website and included in the next CEBC email alert.