CEBC Review and Rating Process
As a new or experienced user of the CEBC website, you might wonder how the CEBC review process works; including how the level of Child Welfare System Relevance is determined, and whether programs can be rated on the Scientific Rating Scale. The review process is described through the links below, as well as in the CEBC Rating Policy and Procedures Manual. The review process typically takes 6 months to complete.
How are Topic Areas Chosen?
The CEBC Advisory Committee is consulted annually and asked to assist in determining the next topic areas that will be added to the website. Once those topic areas are chosen, a topic expert for each area is recruited with the consultation of the CEBC Scientific Panel. CEBC staff works closely with each topic expert to draft a clear definition of the topic area.
How are Programs within the Topic Areas Chosen?
Once the definition of the topic area is finalized, the topic expert generates a list of potential programs for the topic area. At the same time, CEBC staff also conducts an extensive literature search to identify potential programs for the topic area. The lists are then combined and reviewed with the topic expert and a final list is created.
How is the Information about the Program Obtained?
Once the list is finalized, screening questions are sent to each program to:
- Ensure the program has a manual or offers training.
- Establish the legitimacy of the contact’s relationship to the program.
- Collect the program’s goals.
- Obtain the contact’s willingness to fill out a detailed questionnaire about the program.
After the program passes the screening process, a questionnaire is sent out to the program representative to be completed. In addition to the information provided by the program, the CEBC staff also conducts a literature search on each program to obtain any published, peer reviewed research..
How is the Program Information Processed?
For the programs where the program representative completed the questionnaire, CEBC staff reviews and edits what is now considered the program outline. The program representative is contacted for any needed clarification. Published, peer-reviewed articles (i.e., research evidence) that report the outcomes of a research study are summarized and added to the program outline.
What Determines Whether a Program Can Be Rated on the CEBC Scientific Rating Scale?
All outcomes from research studies on a program must have been reported in a published, peer-reviewed journal to be considered by the CEBC staff during the rating process. Please see the Scientific Rating Scale for all of the criteria needed to be rated in one of the five categories on the scale. If a program does not fit the criteria for any of the categories on the scale, it is determined to be NR (Not able to be Rated).
How is an Eligible Program Rated and Evaluated?
Program outlines that are able to be rated are sent to the raters along with the published, peer-reviewed research articles and with a copy of the Scientific Rating Scale. Typically, the raters include the topic expert, and two of the CEBC staff. If there is a discrepancy, the CEBC Scientific Director makes the final decision. In addition to a rating on the Scientific Rating Scale, each eligible program is examined to see which child welfare outcome(s) (e.g., safety, permanency, and/or child/family well-being) are addressed in the research evidence for that program.
How is the Child Welfare System Relevance Level of a Program Determined?
When is the Program Loaded onto the CEBC Website?
The edited program outline is then finalized by adding the Scientific Rating and Child Welfare Outcomes information, if applicable; the Child Welfare System Relevance Level; and any standardized language used by the website. Then it is sent for one last review before the program is loaded onto the website. Once a majority of the programs for a new topic area are ready, then the topic area and new programs are added to the CEBC website and listed in the What’s New section. The rating for all new programs added onto the website is considered provisional for 60 days allowing for comment from CEBC users.
How is a Program Added to Existing Topic Areas?
A program that is brought to the attention of the CEBC either by its users or by itself is taken into consideration by the CEBC staff during an open submission window once a year usually in June. To find out more about this process, please visit the FAQ section. To be eligible to be added to the CEBC, the program must fit into one of the existing topic areas on the website and have either enough research evidence to obtain a rating on the Scientific Rating Scale or be widely used and/or marketed in California. Once it is deemed eligible, the review process is very similar to the one listed above except that the topic expert is usually not one of the raters, instead an additional rater from the CEBC staff is used.
How is a Program Re-Rated?
Once the CEBC staff becomes aware of new research evidence on the program, either through the program’s representative, a CEBC user, or the bi-annual literature review process, the program re-starts the CEBC rating process described above. Once the rating process is finished, the program representative is alerted about the rating change. After that, the program’s information is updated on the CEBC website, it is again under a 60-day provisional rating, and it listed in the What’s New section.
What about Programs that Don’t Respond?
All programs identified by the Topic Expert that meet the CEBC screening criteria (see “How is the Information about the Program Obtained?” above) are highlighted on the CEBC website, even if a completed questionnaire is not received from the program representative. These programs have a note documenting that the program representative did not respond to the CEBC's request for information. Programs which fall into this category have a shortened outline (compiled by CEBC staff from publicly available sources), a Scientific Rating and Child Welfare Outcomes information, if applicable; and the Child Welfare System Relevance Level included in their listing. Program representatives are welcome to complete the full questionnaire at any time.