How are Programs on the CEBC Reviewed?
As a new or experienced user of the CEBC website, you might wonder how the CEBC review process works; including how it is determined whether programs can be rated on both the Scientific Rating Scale and the Child Welfare Relevance Rating Scale.
How are Topic Areas Chosen?
Numerous topic areas are relevant to child welfare; therefore, the CEBC Advisory Committee is consulted annually to determine the five topic areas that will be added to the website during the grant’s next fiscal year. Once those topic areas are chosen, a topic expert for each area is recruited with the consultation of the CEBC Scientific Panel. CEBC staff works closely with each topic expert to draft a clear definition of the topic area.
How are Programs within the Topic Areas Chosen?
Once the definition of the topic area is finalized, the topic expert generates a list of potential programs for the topic area. CEBC staff also conducts an extensive literature search at the same time to see what programs could be in the topic area. The lists are then combined and reviewed with the topic expert and a final list is created.
How is the Information about the Program Obtained?
Once the list is finalized, screening questions are sent to each program to ensure the program has a manual or offers training, to establish the legitimacy of the contact’s relationship to the program, and to obtain the contact’s willingness to fill out a detailed questionnaire about the program. After the program passes the screening process, a questionnaire is sent out to the program representative to be filled out in a timely manner. Once the questionnaire is received back, the research review process begins. If a program does not ever return the questionnaire or refuses to fill it out after initial agreement to, then it is considered a nonresponder and an abbreviated description of the program is created by CEBC staff using publicly available information.
How is the Program Information Processed?
For the programs where the program representative completed the questionnaire, CEBC staff reviews and edits what is now considered the program outline. The program representative is contacted for any needed clarification. Published, peer-reviewed articles (i.e., research evidence) that report the outcomes of a research study are summarized and added to the program outline. For the programs from nonresponders, any published, peer-reviewed articles that report the outcomes of a research study are also summarized. The summaries are then edited and added to the nonresponder program outline.
What Determines Whether a Program can be Rated on the CEBC Rating Scales?
Programs are able to be rated on the Scientific Rating Scale and the Child Welfare Relevance Rating Scale only if they have published, peer-reviewed articles (i.e., research evidence) that report the outcomes of a comparison study.
How are the Eligible Programs Rated?
Outlines for programs that are able to be rated are sent to the raters along with the research articles and with a copy of both scales. Typically, the raters are the topic expert, and two of the CEBC staff. If there is a discrepancy, the CEBC Scientific Director makes the final decision. In addition to a rating on each rating scale, each eligible program has the outcomes highlighted in the research articles examined to see which child welfare outcome(s) (e.g., safety, permanency, and/or child/family well-being) are fulfilled by a child welfare client using the program.
When are the Programs Loaded onto the CEBC Website?
The edited program outline is then finalized by adding the rating information, if applicable, and any standardized language used by the website. Then it is sent with a passive agreement deadline to the program representative (even if he/she is a nonresponder) for one last review before the program is loaded onto the website. Once enough of the programs for a new topic area are ready (usually a majority), then the topic area and new programs are added to the CEBC website and announced in a CEBC email alert.
How is a Program Added to Existing Topic Areas?
A program that is brought to the attention of the CEBC either by its users or by itself is taken into consideration by the CEBC staff. To be eligible to be added to the CEBC, the program must either have enough research evidence to obtain a rating on the Scientific Rating Scale or be used and/or marketed in California. Once it is deemed eligible, the review process is very similar to the one listed above except that the topic expert is usually not one of the raters, instead an additional rater from the CEBC staff is used. The other notable difference is that the program is loaded onto the website when there are enough other programs or new information to warrant sending out a CEBC email alert.
How is a Program Re-Rated?
Once new research evidence is presented to the CEBC, the program re-starts the CEBC rating process described above. However, the raters do not usually include the topic expert, but instead an additional rater from the CEBC staff. Once the rating process is finished, the program representative is alerted about the rating change. After that, the program information is updated on the CEBC website and included in the next CEBC email alert.