Introduction

This supplement to the Swipe Safe Solomon Island Monitoring and Evaluation report presents the refined, working set of online safety indictors developed over three phases of living lab implementation in the Solomon Islands. Qualitative data generated by young people, parents/carers, and community stakeholders in Phase 1 about their ideas, perceptions, and experiences of digital and online technologies contributed to the indicator formulation. Participants’ key challenges and critical issues were identified from that original data. Critical issues were then used to develop an initial longlist of indicators that were further tested and refined in Phases 2 and 3.

A key part of our Young and Resilient indicator development process is condensing a longlist of indicators into a smaller, manageable set of indicators to best address a core theme. The indicators presented here are primarily focussed on online safety, however, a few indicators addressing associated themes (health, infrastructure) have also been included because we judged them relevant to the overall context of digital and online technology access and use. Two further considerations are worth noting:

1. The indicators presented here are child-centred, not child-led. Indicators represent our assessment, based on our expertise and experience, of child and adult participants’ views. In other words, indicators are our interpretation of participants’ preferences, rather than literal translations of those preferences
2. The indicators presented here should be viewed as a living resource. We have designed indicators to be flexible and adaptable for use across different conditions. People and organisations using these indicators can select those most applicable for their aims and refine phrasing and presentation to best suit their application context. Moreover, indicators are also open to revision and evolution. We recommend regular review and critique to ensure indicators remain relevant and applicable to peoples changing digital and online contexts and needs

Finally, our child-centred indicators are designed to complement other indicator frameworks and methodologies, not replace them. Our development process and resulting indicators are purposefully child-centred because existing indicators may omit children’s experiences and aspirations, or privilege adult perceptions and points of view. We have sought to develop indicators that align more with children’s worldviews. That being said, existing indicator frameworks offer valuable resources to help understand issues and can be used in tandem with the indicators herein (see Appendices for a selection of existing indicator and assessment frameworks). The child-centred indicators from this project are presented in Table 1 below. Explanatory information about definitions, meaning, and indicator application follow.

What are Indicators?

We define indicators as observable signs that point to changes over time that can be assessed for an issue or objective. They are useful for describing what something is like right now or assessing if something has changed across time and space (Liebowitz & Zwingel, 2014). In line with other scholars, our indicators are aspirational – they represent the world as we would like it to be; they are signs of an ideal state (McKinnon, Carnegie, & Gibson, 2016, 1387; Law & Urry, 2004, 390).

Indicator Structure

• Outcome: outcomes reference and summarise a critical issue identified in the living lab data. Outcomes represent what matters to children.
• Ideal State: ideal states represent the world as children would like to see it
• Factor: factors represent phenomena that affect the state or condition of indicators. They can be used to identify clusters of indicators/questions
• Aspirational Indicator: aspirational indicators represent things that would need to happen for the ideal state to occur
• Reference: the indicator reference is a unique reference number identifying a specific indicator and its context:

CI- | OS- | XX. | YY. | zz Child-centred indicator | Online safety | Outcome abbreviation | Indicator number | ISO3166-1 Alpha 2 country code

Two-digit indicator numbers are calculated within relevant outcomes. The two-digit ISO suffix reflects the location the indicator has been developed in and adapted for (sb = Solomon Islands). As the indicator set is further deployed and adapted, this suffix should also be updated to identify the relevant contextualisation of a particular indicator set (e.g., by country, region, etc.).

BACKGROUND

One way of thinking about indicators is that they are practical tools to help assess changes in things over time. However, the terminology and process of understanding and using indicators may be unfamiliar to organisations seeking to use them. Accordingly, for indicators to be useful, they require clear guidelines about how they can be used. The following sections describe key components of and considerations for the indicator application process and examples of how indicators could be applied:

• Swipe Safe Concepts & Issues: describes key elements addressed in the Swipe Safe program and their general relationship to child-centred indicators
• Assessing Change: describes the difference between indicators and measures, and a process through which measures can be applied to assess indicators
• Developing Questions: describes the importance of choosing or developing clear and relevant questions to collect data
• Choosing Methodologies: describes the role of methodologies and approaches that questions can be deployed through
• Indicator Process Summary: presents a summary of the indicator application process. Appendices presents examples of what application might look like

Concepts & Issues

The Swipe Safe program uses six key concepts to ground participants’ understanding of the internet and online safety: Public; Permanent; Connection; Anonymity; Sources of Information; Respect. Swipe Safe training sessions further explore and teach knowledge and skills about those concepts via active learning modules related to the following issues: Being Responsible; Scams/Hacking; Who Am I Talking To?; Spending Time Online; Managing Offensive Things; Oversharing; and Cyberbullying. Training sessions include examples of how young people might experience each of these specific issues online (e.g., potential online scams, cyberbullying).

Learning modules develop participants’ thinking, knowledge, and skills in relation to the discrete concepts and issues, and how they interrelate. For example, the idea that the internet allows people to be anonymous is relevant when considering sources of information, while participants may consider the potentially public nature and permanency of online content when thinking about respectfulness to others online. Individually, the concepts address a subset of distinct factors important for keeping people safe online and, when taken together, they establish and reinforce the idea of online safety as an overarching, beneficial state or condition.

Our child-centred indicators are based on data generated in living labs from young people and adults who had completed or were about to complete Swipe Safe training, and so much of the indicator-related data generated in living labs will have been flavoured by participants’ awareness of the six key Swipe Safe concepts and related issues covered in the training modules. However, living lab data generation activities did not themselves focus on those specific concepts or issues. Rather, indicator activities were designed to surface living lab participants’ perceptions and ideas about online safety and digital literacy more broadly. Accordingly, while all child-centred indicators are related to themes relevant to digital literacy and young people’s safety online (and so complementary to Swipe Safe’s overarching ideal of online safety), not all the indicators directly align with specific concepts and issues covered in Swipe Safe training.

Put another way, our child-centred indicators are not exclusively designed to evaluate or assess the effectiveness of the Swipe Safe training program. The body of indicators as a whole are intended to help broader assessment of online safety and digital literacy. Therefore, indicators can be selected and adapted for different training programs and contexts or applied in non-training environments (e.g., to establish current conditions of digital literacy or safety). Moreover, as previously noted, indicators are living resources and so will alter, evolve, and grow in different operational conditions, and as online trends change, and technology develops.

Of course, child-centred indicators can also be used to help assess the Swipe Safe program. Across the range of indicators, those most directly useful for Swipe Safe assessment will be indicators that align to the program’s key concepts or issues (i.e., the specific knowledge and behaviour the program seeks to influence). We have adopted the following definitions for these key elements, derived from Swipe Safe documentation with two amendments, as described, to simplify alignment:

Concepts

• Public: information on the internet has the potential to be accessed by everyone
• Permanent: everything on the internet stays there forever
• Connection: the internet allows us to connect with people and content from everywhere (we have added “content” to broaden the idea of connection beyond people only)
• Anonymity: the internet can allow people to pretend to be someone they are not, and can also make us feel invisible
• Sources of Information: we can access more information than ever before, but we don’t always know where it comes from and if it is true
• Respect: we need to act politely to others on the internet

Issues

• Responsible Behaviour: young people thinking about and acting responsibly online
• Negative Experiences: scams, hacking, cyberbullying, and other harmful content like violence, pornography (we have consolidated scams/hacking and cyberbullying and included harmful experiences more broadly)
• Identifying People: young people thinking about and identifying who they are interacting with online
• Time Online: the time young people spend online
• Managing Risk and Harm: young people thinking about and responding to online risk and harm
• Oversharing: young people thinking about and managing the information they share online

Appendices shows the alignments between child-centred indicators, Swipe Safe concepts and issues, and other overarching online safety and digital literacy themes. The general process to use indicators for Swipe Safe assessment involves:

1. reviewing the indicators for their alignment to Swipe Safe concepts and themes
2. identifying a subset of the indicators that best address the knowledge and/or behaviour change to be assessed
3. devising questions that match indicators and that will resonate with participants
4. deploying questions to gather data about indicators

IMPLEMENTATION

Our child-centred indicators are not measures; rather they are signs that can be measured or assessed in some way. Indicators are operationalised through assessment. Assessment of change in an indicator can involve several steps including, for example:

1. identifying a relevant, measurable factor and associated question(s) for selected indicators (e.g., data from which will provide information reflective of the state of an indicator)
2. deciding what method best suits the question(s) and likely participants method (e.g., qualitative, quantitative, a question, survey, group activity)
3. applying the method (e.g., asking the question, administering the survey, running the activity)
4. collecting and interpreting the data
5. re-assessing the indicator at a future time and comparing data to see if a change in indicator state has occurred

Developing Questions

It is possible to use the formal phrasing of aspirational indicators as assessment questions. For example, for indicator CI-OS-SE.01.sb, a scale question could use the prompt “Children say the behaviour they see from other people online is not harmful to them or others” with response items “strongly disagree” to “strongly agree”. However, this question formulation may not be the most accessible to end users. In testing during living labs, plain language examples of questions reflective of indicators resonated and were more easily understood by both young and adult participants. Thus, for CI-OS-SE.01.sb, operationalising the question as “Is most of the behaviour children encounter online safe for them and other children?” may be a more effective construct.

The most accessible and effective question constructs will vary from context to context and so must be formulated by the end users on the ground who are selecting and applying the indicator set and who are experienced and expert working with their likely participants. A non-exhaustive, indicative list of possible plain language question formulations for each indicator is presented in Appendices. People or organisations applying the child-centred indicators can consider and adapt these formulations or choose to develop their own.

Choosing Methodologies

As with the questions selected to assess indicator state, the methodologies through which questions will be delivered will vary based on what is practicable for those applying the indicators and accessible and straightforward for likely participants. We favour using a mix of both qualitative and quantitative methodologies as the best way to collect comprehensive and deep data about indicators. For example, focus groups or living lab workshops where participants complete individual and small-group activities can incorporate rich data collection (e.g., interviews, discussions) and basic quantitative techniques (e.g., quick polls). On the other hand, while mixed-methods collection offers the most complete data generation options, end users may not have the time or capacity to implement such designs, and so may opt for either qualitative (e.g., brief interviews) or quantitative (e.g., surveys, brief polls) options. Using mixed-methods data collection for indicators is optimal but not essential. Either qualitative or quantitative designs can be implemented on their own to collect data relevant to indicators. The specifics of implementation will again depend on the capacity and preferences of end users, however, the indicative question formulations in Appendices can be adapted and applied for qualitative, quantitative, or mixed methods approaches. End users may already have a preferred data collection methodology or have access to guides to assist their selection of methodology. Information and resources to help choice and application of data collection methods is also available online from the Young and Resilient Co-Research Toolbox.

Using Indicators

Our child-centred indicators are designed to help assess outcomes (e.g., whether the state of critical issues relevant and important to young people change over time). Indicators can be applied to assess specific programs over time (e.g., Swipe Safe) or to gain an understanding of the broader online safety environment (e.g., to surface young people’s current experiences and perceptions of online literacy or safety). While the general process (summarised below) of applying indicators towards either goal is the same, at key steps in the process decisions based on goals and context will help determine any final implementation strategy. The nature of an implementation will also depend on the preferences and capacities of the organisations applying the indicators: for example, available resources, preferred modes of engagement with participants or end users. Put another way, although we can summarise the implementation process, it is not meaningful to give guidance that leads to a single, definitive mode of implementation because implementation will vary between use cases. To illustrate and add clarity to the summary process, Appendices presents some examples of what different implementations might look like in different contexts, however, organisations themselves will ultimately determine and design how their own application of indicators unfolds.

Reminder: Indicators are the things you will assess. Factors identify indicators/questions that address similar phenomena. Questions are the things you will ask. Methodologies are how you will ask your questions. Materials are the items that support you asking questions. Data is the responses to your questions. Analysis is how you make sense of your data.

To use indicators in an applied setting, follow these steps:

1. Review Indicators: review the overall working set of child-centred indicators to familiarise yourself with their content, and the outcomes and ideal states they represent
2. Choose Indicators: select a subset of indicators that best align to the outcomes and states you are interested in
1. we strongly recommend selecting a subset of indicators (e.g., 4 to 10 indicators) because: 1) the working list of indicators cover diverse issues and outcomes and so a smaller subset will likely provide better focus on the issues that matter most to you, and 2) a smaller subset will be easier to operationalise and manage for practical data collection
2. the size of your indicator subset will also be influenced by your operational resources and capacity, and the context of your deployment (e.g., the sources – participants, institutions, etc. – you will be collecting data from)
3. for assessing the Swipe Safe program, it may be most productive to choose the bulk of your subset from indicators that align more directly with Swipe Safe concepts and issues (see Appendices). However, you can also include indicators that are not specifically aligned if they represent useful outcomes and ideal states
4. for assessing digital literacy and online safety broadly, your indicator subset can be drawn from the overall working set, including Swipe Safe-aligned and non-aligned indicators
3. Select Factors (optional): identify factors associated with your selected indicators to help filter the questions you will ask (complete this optional step to further narrow the list of questions you want to consider)
4. Pick Methodology: decide on the methodology to engage with your participants and deliver your questions to them. Methodologies include qualitative and quantitative options like workshops, interviews, focus groups, online/face-to-face surveys, quick polls, take home activities, etc. If you have a preferred methodology, tailor and incorporate your questions accordingly. Alternatively, a wide range of open-access methodology guides are available online (including from the Young and Resilient Co-Research Toolbox ). Examples are also included in Appendices. You should choose a methodology that will generate data you are confident you can analyse. When deciding on methodology, consider how you will make sense of the data you will collect
5. Devise Questions: determine what questions you will ask to gather data for each of your indicators. Examples of plain language question options have been listed alongside indicators. You can select and/or adapt questions from this list or devise your own formulations that will be most relevant to your goal and effective for your participant base. You can revise and sharpen questions (e.g., based on the decisions you make about methods etc.) until you are ready to implement
6. Prepare Materials: create or gather the supporting materials you will need to ask questions. Specific materials will vary based on the methodology you have chosen. For example, an online survey will require a survey hosting platform and, potentially, mobile devices for in-person delivery; printed copies will be needed for a face-to-face survey; a workshop will require activity descriptions and printed or online worksheets, etc.
7. Collect Data: plan, organise, and deliver your materials. Depending on your selected methodology, this may include random or targeted delivery of quantitative paper-based or online surveys; planning and running mixed-methods face-to-face or online workshops; including brief quantitative poll-type activities within existing programs agendas, etc.
8. Analyse Data: the type of analysis you undertake will be determined by the nature of the data you have collected. Plan well ahead of time how, when, and what resources you will need to make sense of your data
9. Repeat Data Collection (optional): indicators are primarily intended as tools to help assess changes over time. For that to happen effectively, data must be collected and compared from (at least) two different times. If one of your goals is to assess change, then data collection (Step 7 above) should be repeated. Ideally, the same indicators assessed during original data collection will also be assessed during repeat collection and the same participants will be recruited to generate the data. Repeating equivalent methodologies will simplify data comparisons (e.g., comparing quantitative survey data with quantitative survey data, or qualitative interview data with qualitative interview data). Using the same or similar questions at different times can also streamline and simplify data collection and comparison. However, repeat data collection may not be required if your principal goal is simply to understand the current state of a phenomenon (e.g., to surface perceptions of online safety at a specific point in time)