Any dissemination of results must indicate that it reflects only the author's view and that the Commission is not responsible for any use that may be made of the information it contains H2020-SC6-REV-INEQUAL-2016 netWorked Youth Research for Empowerment in the Digital society Grant Agreement number: 727066 Framework Design for WP7: 1st Cycle Evaluations Anna Renau Gani Naylor PYE Start date:01/05/2017 Due date: 30/06/2017 Any dissemination of results must indicate that it reflects only the author's view and that the Commission is not responsible for any use that may be made of the information it contains H2020-SC6-REV-INEQUAL-2016 EVALUATION AND INTERPRETATION PHASE (WP7) Workpackage 7 focus on the Evaluation and Interpretation of the research activities. The evaluation of the research activities will work at various levels. Firstly, it is necessary to evaluate whether the social dialogue and the collection of research activities is functioning appropriately, whether the toolkit is appropriate, and whether the overall set of results is useful. Secondly, it is necessary to evaluate each particular research activity according to its own design and success criteria. Lastly, it is necessary to evaluate which of the results, which combinations of the results of the cycle constitute or include insights that may be relevant for policy and susceptible to presentation as recommendations. In particular, it is likely that insights that are common across a range of research activities across Europe may be generalizable to recommendations. This stage of the WYRED cycle focuses on the three levels. As the WYRED project works in cycles, this work package has two different work periods, and in the second cycle may be subject to adjustments in the light of the results of reflection on the first cycle of evaluation. However, in addition to this conventional evaluation, it is also necessary to make the results accessible to other constituencies than the research/policy community. This requires interpretation of the results in other formats, using other more creative, informal or dynamic approaches. In some senses the artefacts generated in WP6 will have the potential to do this, but interpretative processes are also envisaged in this work package. 1st cycle Social Dialogues Particular research activity Collection of Research activities Tool kit Results – what results do we want? Design appropriate? Appropiate tools? Are criteria successful? PARTNERS SELF-ASSESSMENT PARTICIPANTS VALORIZATION Any dissemination of results must indicate that it reflects only the author's view and that the Commission is not responsible for any use that may be made of the information it contains H2020-SC6-REV-INEQUAL-2016 2nd cycle Results Compile common aspects from EU participants -> policy recommendations EVALUATION What are we evaluating: In this WP7 we will focus to evaluate the project activities proposed to our participants and the results collected form their execution by the participants themselves. It aims to identify whether the activities and tools used through the project are appropriate and are providing the desired results for the project. It also envisages guiding young people to identify the values and importance of their researches. Themes: Digital Society Topics Priority Outcomes Youth areas of interest Impact expected from our actions/activities How do we evaluate our projects: One of the basic thing to start to evaluate our projects is to ask the proper questions: • Objectives of the project/programme/activities: what do we want to achieve • How the project is going • How participants are feeling • What might need to change Structure of an evaluation process: 1. Identify the right questions we want to ask in order to get the information we need or want 2. Using a proper Tool we will stablish a research method to be implemented by all participants 3. Once participants have done all evaluation exercises/activities, it is important to gather data in a standardised way, easy to classify, understand and interpret. 4. A report will be done with all information provided after analysing al data collected. Right Questions Research Method Gather Data Report Conclusions -> Recommendations Any dissemination of results must indicate that it reflects only the author's view and that the Commission is not responsible for any use that may be made of the information it contains H2020-SC6-REV-INEQUAL-2016 5. From the report, it will be possible afterward to get specific information that will allow us then to provide some recommendations for a formal report to the Commission. WYRED Objectives: • Involve Young People in Social Dialogue -> their voice • Participatory Research -> self expression • Talk about Digital society -> young’s attitudes, norms & values • Diversity -> wide range of stakeholders • Moving target -> young grow up and develop, so it will be an ongoing project with different participants Source: New Opportunities for Innovative Student Engagement (NOISE) program; Youthrex.com SAMPLES TO BE DEVELOPED FOLLOWING THESE MAIN QUESTIONS AND PARTNERS INPUTS • Structure / content / objectives: to know if it is appropriate and participants are interested. Are we going in the right direction to achieve WYRED objectives? • Feelings of participants / learning process: to know how engaged are participants and IF WYRED is providing them an added value > Knowledge about DS before/after project > Valorisation of activities (interest): content / length / useful > Initiative to research > Participation from in person sessions to platform > Changes in attitude – did participants change their concern about one specific topic > Changes in behaviours – did participants take action after participating in WYRED activities • Next steps of the project: to know their interest on the project and if they are thinking about continuing, recommend it… Any dissemination of results must indicate that it reflects only the author's view and that the Commission is not responsible for any use that may be made of the information it contains H2020-SC6-REV-INEQUAL-2016 Format Evaluations for young participants and stakeholders would be slightly different: • Children up to 11 year old: questionnaire with pictures – each picture will correspond to a specific item • Young people up to 17: questionnaire with sentences and click options • Older than 17: questionnaire with sentences and images *The format, images and presentation of the questionnaire can be adapted in different creative ways. **Each country/partner will have to adapt the sentences/images to be as much accurate and close to the concept/idea we are asking for Structure of evaluation session – 1 hour¿? 1. Set up and introduction for the evaluation activity 2. Evaluation process a. Assistance of participants if anyone have questions and clarify i. Give examples and clarifications as needed ii. Share all participants questions, comments and inputs with the groups iii. Let the YP propose new things 3. Each partner collect all evaluations and translate them if necessary into English – all partners 4. Collect all data and extract results – all partners 5. Analyse results and create report – PYE and BOUNDARIES (partners feedback & comments) Evaluation tool kit Toolkit to be developed with specific examples. It will contain: • Evaluation session layout • Examples and evaluation grid that participants can use Any dissemination of results must indicate that it reflects only the author's view and that the Commission is not responsible for any use that may be made of the information it contains H2020-SC6-REV-INEQUAL-2016 EVALUATIONS OF THE SOCIAL DIALOGUES AND RESEARCH ACTIVITIES FOR YOUNG PEOPLE Objectives: • Self-administered evaluation processes • Children and young people will be active in this process choosing the messages, formats and channels they see as most appropriate to present their results • Help young people to identify the value and relevance of what they are doing in WYRED activities The importance of using creative expression approaches: Creative and arts based approaches can be particularly powerful, especially at the data collection and dissemination phases of the evaluation cycle. They include a vast array of techniques including photography, film and visual arts, poetry and creative writing, music, drama and performing arts. These are used in order to uncover hidden perspectives, add empathic power, and strengthen participants’ voices CREATIVE AND ARTS BASED EVALUATION METHODS Written by Norma Daykin with the Creative and Credible Project Team SAMPLES TO BE DEVELOPED FOLOWING THESE QUESTIONS In all research activities YP should identify WHY did they choose that specific topic, WHAT did they discover and WHAT exactly the want to communicate and HOW they will explain it to the public. - Why do you want to talk /explain/research about this specific issue? - Why should others should learn/know about this? > It helps to keep in mind the main topic of the research (normally it would come from the selection of the Delphi Research main topics or Social Dialogues) > It tells why the issue/topic is important for them - What is the specific theme/issue you have been working on, or you have participated in? - What was your level of understanding before the research? And after? > It allows them to identify what they are learning and realize about their new learnings/understandings - What have you discovered/learnt? – results: ▪ Numbers? ▪ Areas of interest? ▪ Young people/adults behaviours? ▪ Actions or activities? ▪ Organizations working this specific field? ▪ Etc. - What was your mood during your research? And when explaining your results? Any dissemination of results must indicate that it reflects only the author's view and that the Commission is not responsible for any use that may be made of the information it contains H2020-SC6-REV-INEQUAL-2016 > Level of motivation and passion to learn more about something they are interested in - How can you present you project/research: - To whom are you explaining it - Time: how long do you need to explain - Format: creative ways: film, photography, music, creative writing, visual arts, performing arts, etc. > To identify the support that suits better to everyone we can ask: what are your hobbies? Or what activity do you like the most? Etc (dynamics or exercises can be organized to help YP to identify the support) REFERENCES García-Peñalvo, F. J. (2016). The WYRED Project: A Technological Platform for a Generative Research and Dialogue about Youth Perspectives and Interests in Digital Society. Journal of Information Technology Research, 9(4), vi-x. García-Peñalvo, F. J. (2017). Project status overview: 2nd Project Meeting. Retrieved from https://repositorio.grial.eu/handle/grial/857. García-Peñalvo, F. J. (2017). WYRED Platform. Initial functionalities. Retrieved from https://repositorio.grial.eu/handle/grial/859. García-Peñalvo, F. J., & Durán-Escudero, J. (2017). Interaction design principles in WYRED platform. Paper presented at the HCI INTERNATIONAL 2017, Vancuver, Canada. García-Peñalvo, F. J., & Kearney, N. A. (2016). Networked youth research for empowerment in digital society. The WYRED project. In F. J. García-Peñalvo (Ed.), Proceedings of the Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM’16) (Salamanca, Spain, November 2-4, 2016) (pp. 3-9). New York, NY, USA: ACM. doi:10.1145/3012430.3012489 García-Peñalvo, F. J., & Kearney, N. A. (2016). Presenting the WYRED project at TEEM 2016 Conference. Retrieved from http://repositorio.grial.eu/handle/grial/731. doi:10.5281/zenodo.208223 Gojkovic, T., & Chatzimichail, P. (2017). Networking Report Advances. Retrieved from https://repositorio.grial.eu/handle/grial/860. Nametak, S., Cicala, F., & Taccetti, G. (2017). WP8 Valorisation Advances. Retrieved from https://repositorio.grial.eu/handle/grial/864. O’Reilly, M. (2017). WP5 – Social Dialogues. 2nd Stage of the Research Cycle. Retrieved from https://repositorio.grial.eu/handle/grial/861. Soffer, T., & Hauptman, A. (2017). WYRED results from first round Delphi survey: Young people & Stakeholders. Retrieved from https://repositorio.grial.eu/handle/grial/862. WYRED Consortium. (2017). WYRED Research Cycle Infographic. European Union: WYRED Consortium. Retrieved from https://repositorio.grial.eu/handle/grial/839. doi:10.5281/zenodo.572622 WYRED Consortium. (2017). WYRED Research Cycle Overview Infographic. European Union: WYRED Consortium. Retrieved from https://repositorio.grial.eu/handle/grial/840. doi:10.5281/zenodo.572633 Any dissemination of results must indicate that it reflects only the author's view and that the Commission is not responsible for any use that may be made of the information it contains H2020-SC6-REV-INEQUAL-2016 WYRED Consortium. (2017). Requirements Document (WP3_D3.1). European Union: WYRED Consortium. doi:10.5281/zenodo.292978 Yılmaz Doğan, Y., & Arati, D. (2017). Framework Design for WP6: Activity Tool Kit. Retrieved from https://repositorio.grial.eu/handle/grial/863.