1. Access to Justice

Access to justice is a problem in Canada. While debates exist about what exactly the problem is, no one doubts that many Canadians struggle to have their legal needs properly met. Indeed, the data in support of this conclusion are overwhelming.Footnote ¹

This study focuses on access to justice in a particular context: family law. A striking feature of contemporary family law litigation in Canada is the large number of SRLs in court. Justice system actors have estimated that between 50 percent and 80 percent of parties in Canadian family cases are self-represented.Footnote ² Empirical studies of the SRL phenomenon in family law are consistent with these anecdotal impressions.Footnote ³

In recent years, the legal system has started to respond to the in-court challenges that SRLs face. Judges and, to a lesser extent, opposing counsel are now recognized to have explicit obligations to facilitate fair hearings for SRLs.Footnote ⁴

But SRLs also face serious barriers before they even step into a courtroom. In multiple studies, SRLs have identified complex court forms as a particular “pain point” in their justice journeys.Footnote ⁵ In one national study, 97 percent of court staff surveyed agreed that SRLs required help in completing court forms.Footnote ⁶ With respect to family law proceedings, more than 50 percent of responding litigants in one 2005 study reported that they “found difficulties with the court forms and knowing their legal rights.”Footnote ⁷ In another 2012 study, several SRLs who are expressing frustration with family law forms are quoted.Footnote ⁸ For example, one respondent in this study stated that “the forms you are given to fill out are extremely difficult to understand. They are designed for lawyers to fill out and not regular people [re]presenting themselves.”Footnote ⁹

While there has long been awareness that court forms cause difficulties for SRLs, only recently have there been studies about what makes court forms complex for non-legally trained people. Much of this research has focused on linguistic complexity (i.e., reading difficulty).Footnote ¹⁰ A different approach has been taken by two of the authors of this article, Jacquelyn Burkell and Amy Salyzyn, who have led previous studies of court forms using a “functional literacy” framework.Footnote ¹¹

2. The Functional Literacy Framework

The concept of functional literacy moves beyond more traditional assessments of literacy, which tend to focus on reading level, and looks to individuals’ ability to understand and use information to complete tasks.Footnote ¹² Functional literacy is relevant, for example, to whether a person can understand how much medicine to take after reading a prescription or to effectively apply for government benefits by filling out an applicable form.

Given this task focus, a functional literacy framework can be used to evaluate why SRLs may find court forms complex. To demonstrate this point, we (Burkell and Salyzyn) have offered the example of a prompt on a court form that asks for an address.Footnote ¹³ While many people will understand what “address” means in a dictionary-definition sense, it may still be difficult for them to answer this prompt, depending on whether the form indicates whose address to include (e.g., is it the user’s address, the opposing party’s, or the court’s?) and how difficult it is to find the address needed (e.g., it will be easier to give one’s own address than to find a courthouse address). This very simple example helps to show why readability alone is not a complete measure of court form accessibility.

Implicit in a functional literacy framework is the notion that there are two avenues of intervention available when an individual cannot complete a task: (1) training to improve literacy skills, or (2) redesign of tasks to decrease complexity. Building on the latter, in 2005, Julien Evetts and Michel Gauthier published a guide for assessing task complexity that can be used to assign a literacy-level requirement to tasks and, in the process of doing so, identifies what features of the task are likely to make it difficult for users to complete.Footnote ¹⁴

Evetts and Gauthier’s task complexity assessment system is complex and a full explanation is beyond the scope of this article. By way of an example, one thing that their system examines is strategies for matching information. This means considering what a user needs to do in order to match given information (as contained in a prompt) with requested information (a correct or otherwise appropriate answer).Footnote ¹⁵ The Evetts and Gauthier system acknowledges that a user’s difficulty with a task can be affected by whether a user needs to use a “locating,” “cycling,” “integrating,” or “generating” strategy.Footnote ¹⁶

For example, a court form prompt that requires a user to fill in and thus “locate” their landlord’s address is less complex than a prompt that asks the user to explain how they came up with a particular rent abatement (this would be a “generate” task). “Cycling” refers to “the process of making multiple matches to retrieve multiple pieces of information.”Footnote ¹⁷ Having to flip back and forth between a court form and a guide to filling out the court form would be an example of “cycling”; needing to cycle can make tasks more difficult to complete. Finally, “integrating” refers to “the process of contrasting and comparing information once it has been identified through cycling.”Footnote ¹⁸

In addition to examining information matching strategies, Evetts and Gauthier’s system also looks at other factors, such as the nature of the information requested (e.g., is it highly concrete or abstract?), whether inferences are needed, and the number of pieces of information that need to be included.Footnote ¹⁹ The presence of “plausible distractors” (a term of art referring to broad descriptions or vague requests in prompts that could give rise to incorrect answers) is also recorded.

In an earlier 2016 study of court form complexity, we used the Evetts and Gauthier system to evaluate the complexity of over 280 tasks contained in four different court forms used in Ontario.Footnote ²⁰ A second study, published in 2019, examined the perspective of human subjects regarding court form complexity, with the goal of assessing whether the quantitative analysis generated from the first study might be reflected in actual user experience.Footnote ²¹ In large part, the results of our second study confirmed the evaluation from the first study, although there were some discrepancies (i.e., there were some tasks that were not flagged as potentially complex in the first study but caused difficulty for multiple participants in the second study).

Two insights from these studies are particularly relevant to the research study described in this article. First, we concluded that the use of an instrument that is grounded in a functional literacy approach is a promising way to study court form complexity but that, optimally, this approach should be augmented by user-based testing of task difficulty. Complementary user-based testing can help mitigate both (1) researcher-generated limitations (e.g., preexisting expertise) and (2) limitations inherent in the task analysis tool itself (e.g., certain sources or types of complexity that are not being captured by the tool’s approach). Second, both studies suggested that digital interactive forms in which tailored, detailed, and context-specific assistance can be provided to users (i.e., Guided Pathways) could be helpful tools to reduce court form complexity.

Our study builds on our prior work to evaluate Guided Pathways as a person-centred access-to-justice intervention. Were we correct that digital court form Guided Pathways can be access-enhancing measures? And, if so, how can such Pathways be optimally designed and evaluated?

This study was conducted in partnership with CLEO and focused on CLEO’s Family Law Guided Pathways. These Pathways, developed in partnership with the Ministry of the Attorney General, are online, interactive tools that are free for public use. The Pathways take users through a series of questions and then use those questions to populate the relevant court forms.Footnote ²²

Our goal in studying these Pathways was twofold. First, we sought to determine whether a functional literacy framework, demonstrated in our earlier research to be useful in studying and reducing the complexity of paper or PDF-based court forms, could also be useful when engaging with digital court form Guided Pathways. Second, we sought empirical evidence to support the suggestion in our prior work that digital Guided Pathways can be used to reduce court form complexity for SRLs.

This study had three components. First, the Evetts and Gauthier task complexity assessment system was used to review and revise the Pathways. This part of the study aimed to determine whether a functional literacy framework could be helpful in designing and evaluating court form Guided Pathways. Part II below outlines how CLEO successfully used the Evetts and Gauthier system to improve their Pathways, leading us to answer this part of the study in the affirmative.

Following these revisions, we explored the usability of the Guided Pathway for Form 8. User experience was examined through (a) an analysis of actual (anonymized) information provided by SRLs who have used Guided Pathway for Form 8 and (b) “think-aloud” interviews in which members of the public without legal training were asked to “walk through” the Guided Pathway for Form 8 and also the comparator Fillable-PDF Form 8 using facts in fictional scenarios. Parts III and IV of this article describe these parts of our research in more detail and provide the supporting evidence for our conclusion that Guided Pathways can meaningfully improve court form usability for SRLs.

3. Person-Centred Justice

In its subject matter, design, and execution, this study embraces a “person-centred justice” approach, focusing on those experiencing justice needs and problems as opposed to formal institutions and systems.Footnote ²³

The subject of this study—a CLEO Guided Pathway—aims to be a “person-centred” access-to-justice intervention. While official court forms are designed with justice officials—such as lawyers, court clerks, and judges—in mind, Guided Pathways are tailored-made for persons who are navigating the court system without direct help from legal experts. More specifically, CLEO’s Guided Pathways align with several recommendations in the Organisation for Economic Co-operation and Development (OECD)’s Recommendation of the Council on Access to Justice and People-Centred Justice System, including calls to ensure that justice services are “designed with people at the centre” and are “provided in clear, plain and inclusive language and manner, avoiding complexity.”Footnote ²⁴ The digital nature of the Guided Pathways also aligns with the recommendation that “justice [be made] within reach for everyone regardless of their geographical location, including rural and remote areas, promoting mobility to bring the justice and legal services directly to the people.”Footnote ²⁵

A key part of a person-centred approach is evidence gathering: “people-centered justice services must constantly strive to improve through evaluation, evidence-based learning, and the development and sharing of best practices.”Footnote ²⁶ As described in further detail below, this second and third parts of this study were designed to gather evidence about how users experience the Guided Pathways. This aligns with the specific call in the OECD Recommendations for justice-related services “based on empirical understanding of [people’s] legal and justice needs, preferences and capabilities.”Footnote ²⁷

1. Methodology

This work involved a series of in-house workshops at CLEO that engaged a multidisciplinary team comprising lawyers, design analysts, law and computer science students, and an independent scholar.Footnote ²⁸ First, difficult tasks in the selected Pathways were identified by using complexity factors that were identified by Evetts and Gauthier. Second, changes were made to the selected Pathways with the aim of addressing issues that were identified at the first stage. Third, the team assessed whether the same or similar changes should be made to other parts of the selected Pathways or to different Pathways that were not being studied specifically.

2. Results and Observations

The CLEO review of the Guided Pathways led to multiple targeted revisions. Details of these revisions can be found in a report that was prepared by CLEO and published on its website.Footnote ²⁹ In reflecting on the work conducted, the CLEO team concluded that using a task-based or functional literacy approach “is a good fit for reviewing complexity, especially in the context of automated legal tools […] [given that] the pathways are fundamentally task based.”Footnote ³⁰ In reaching this conclusion, they acknowledged that this exercise did require a significant investment of time, including learning the Evetts and Gauthier framework.

The CLEO report also comments on the value of carrying out task-based assessments of court form complexity by using “multidisciplinary teams in conjunction with a system of user feedback.”Footnote ³¹ The report notes that “[t]he CLEO team reviewed email inquiries from users and feedback from frontline workers and others who help people fill out court forms.”Footnote ³² This observation echoes our observation in our 2019 study that, optimally, a functional literacy approach should be supplemented with user feedback and testing.

Finally, the CLEO report emphasizes the ongoing nature of court form improvement, observing that “[c]ontinually reviewing our pathways to make improvements provides benefits to users […] [and] enables us to identify principles and develop best practices to help us enhance our online legal assistance tools over time.”Footnote ³³

Although this first part of our study was focused on a specific set of court form Guided Pathways, the results have broader applicability. Not only was the use of a functional literacy framework validated as a productive assistive measure when designing court form Guided Pathways, but this research also provides support for concurrent user testing and continual review. These insights can be used by others to optimally design and deploy court form Guided Pathways in different contexts.

1. Methodology

Form 8 is the form used to ask the court for decision-making responsibility (called child custody at the time of the study), parenting time (called child access at the time of the study), child support, spousal support, property division, other family law orders such as a restraining order and/or a divorce together with any other orders dealing with the issues above.Footnote ³⁵

We chose to study the Form 8 Pathway for several reasons. Form 8 is a commonly used family law form that applies to a wide range of potential remedies. Additionally, Form 8 was one of four court forms that we investigated in our 2016 quantitative study by using a functional literacy analytic framework.Footnote ³⁶ Below, we use our prior task-based quantitative analysis of the Fillable-PDF version of Form 8 at several points to provide further insights and comparator information in relation to the data collected in this study on the corresponding digital Guided Pathway.

In this second part of our study, the data used to study the Form 8 Pathway were the anonymized aggregated data from users who interacted with this Pathway between September 2019 and June 2020, after CLEO undertook the functional literacy revisions described in Part II above.

The data included the region in which the user was located (based on a proxy measure of the courthouse selected), the user’s age (in an age-range band), their relationship type (e.g., married or common law), type of claims made, employment status, how many times the user engaged with the Pathway, and, if the user engaged more than once, at which part of the form they exited and whether the Pathway was completed.

One limitation to our analysis was that the data collected included only those users who had made “clean” or “orderly” exits from the system, thereby excluding users who did not go through the formal exit process (e.g., closed their tab/browser simply or shut off their computer). Among this group of users could be those who: browsed the Pathway without entering any data; exited the Pathway at the prompting of questions that were designed to filter out users whose cases were not suitable for the Pathway; exited because of challenges related to the complexity and emotional difficulty of family law issues; experienced technical issues; or became frustrated while interacting with the system. This last group in particular could include users who were experiencing the greatest difficulty with the system but our data would allow no insight into their experiences. Further, the limited data available from users who made “clean” exits did not include some information that would have been interesting to analyze but was not captured by the system, including: (1) total time to complete the form: (2) which, if any, pop-up definitions were used; and (3) at which question a user exited (the data collected only indicated the section of the Pathway that a user exited).

To supplement these data, three survey questions were added at the end of the Guided Pathway:

1. 1. “What part of the Pathway was most helpful?”

2. 2. “What part of the Pathway was most difficult?”

3. 3. “Enter anything else you want to tell us about using the Pathway”

Responses to these open-ended questions provided additional context for the data that were collected, allowing deeper insight into user experiences.

2. Results and Observations

Several conclusions can be drawn from the “clean” exit data.

First, there was significant use of this online tool. Over the data collection period, 1,162 new users (who made “clean exits”) engaged at least once with the Form 8 Guided Pathway. Actual users over this time period would have been an even higher number, given, as explained above, that the data collected did not include anyone who failed to make the standard exit from the Pathway.

Second, there was province-wide use of the tool. The majority of users (75%, or 871) identified their region as being Ontario, with slightly over half (52.1%, 454) from the Central Ontario region (Central East, Central South, or Central West), 16.2 percent (141) from the Eastern region, 9.1 percent (79) from the Northern Ontario region (Northeast, Northwest), 14.7 percent (128) from the Southwest region, and 7.9 percent (69) from the Toronto region.

Third, the tool appeared to be relatively easy for many to use. A surprisingly large proportion of users who made “clean” exits (927, or 79.8%) completed the form and most of those who completed the form (607) did so in only one session (i.e., they did not leave the system, save their work, and return to do more work later).Footnote ³⁷ Although we cannot determine the proportion of total users that this represents, the fact that hundreds of users were able to navigate the system with apparent ease is positive.

Fourth, challenges experienced by users when interacting with the Pathway seemed to occur when they were providing information about legal claims. Among the 320 users who required more than one session to complete the form, the large majority (88.8%, 284) exited where information about the claim was required (Exit 7).Footnote ³⁸ This was also the most common exit point for the 102 users who engaged in more than one session but did not eventually complete the form (51%, or fifty-two users). The questions asked in this section depend on the user’s circumstances and the claims advanced. For example, someone claiming spousal support would be asked about the amount they wished to claim and also about relevant background information, including employment, childcare arrangements, or medical conditions. It is not surprising that users who exited most often left while completing the section relating to claims. Prior to this part of the Pathway, the tasks include: (1) identifying claims to be pursued; (2) adding biographical information about themselves and their spouse; and (3) basic details about the marriage. In our previous 2016 quantitative study of the Fillable-PDF Form 8, the fields that required the user to provide biographical information and basic details about the marriage were ranked as having low complexity, given that the information elicited was highly concrete and was likely to be very accessible to the user (i.e., could be elicited from either memory or easily available personal documents).

A total of 227 users completed one or more of the survey questions, and these data yielded additional insights. In large part, those responding to the survey expressed satisfaction with the Pathway’s usability. By far the most common responses to the question “What part of the pathway was most difficult?” were variations on either “none” or “nothing.” Similarly, the most common responses to the question “What part of the pathway was the most helpful?” were variations of “all of it” or “everything.”

Several survey responses supported our previous prediction that interactive online forms would be easier for members of the public to navigate. For example:

Users also identified specific design features as reducing complexity:

• • Multiple users lauded the “step-by-step instructions” as particularly helpful. As noted in one response: “the organized questionnaire broke down each step and made it a lot easier and less overwhelming to fill out.”

• • Users also specifically complimented the hyperlinked definitions and the “learn more” boxes.

• • Survey responses highlighted the plain language used in the Pathway. For example, one response complimented the helpfulness of “the way each question is broken down in simple terms without all the legalese” and another praised “how legible the questions read so straight forward and not overwhelming causing panic and anxiety.”

At the same time, some user surveys also identified challenges when using the Pathway:

• • One set of challenges relates to the underlying complexity of the family law process. For example, some users expressed difficulty in “locating” requested information—multiple users identified, as a specific challenge, needing to remember dates that the Pathway asked for.

• • Some users expressed difficulty when it came to questions for which they had to “generate” information (i.e., provide an explanation), such as:

  • “Where I have to write examples of what I do. Like caregiving. I’m not sure what kinds of things to write”

  • “The open information areas. Typing space is small. Also vague on just how much one should really say.”

  • “There should be more of an explanation as to what to answer and what not to put in each area.”

• • The overall length and the time it took to complete the Pathway were identified as challenges by some.

• • One user identified the “unexpected emotional toll” of the process as being the most difficult part of the process.

In summary, the survey data appear to confirm the value of a Guided Pathway approach to filling out a court form while also underscoring an observation in our 2016 study that the underlying complexity of the family law system limits the ability to redesign forms in order to remove all the barriers experienced by SRLs.

1. Methodology

Participants were randomly assigned one of four hypothetical scenarios and were asked to complete either the Fillable-PDF format or the Guided Pathway version for Form 8. The participants comprised nineteen individuals who were recruited from a pool of undergraduate students who were receiving academic credit for participating in research studies. The education level of these participants is consistent with the approximately two-thirds of SRLs reported to have university degrees or college diplomas.Footnote ³⁹ Law students were excluded, as were any individuals with significant legal training (i.e., anyone who was a lawyer or paralegal). Participants were required to be fluent in English.Footnote ⁴⁰

Participants were asked to verbalize their thoughts as they completed either the Fillable-PDF Form 8 or the Guided Pathway using their assigned fictional scenario. This concurrent think-aloud protocol, which is a widely employed method for studying the usability of tools or interfaces,Footnote ⁴¹ “has high face validity, since the data obtained reflect the actual use of an artefact, and not the participants’ judgements about its usability.”Footnote ⁴² Participants also engaged in a debriefing interview in which they reflected on their experiences while completing the form.

We considered recruiting individuals with active family law matters as study participants, since, as observed by Steenhuis et al., “the most reliable way to improve a form is to conduct an observational study of real self-represented litigants completing the form and then to identify areas where litigants experience difficulty.”Footnote ⁴³ Ultimately, however, ethical concerns mitigated against this option. Actual users would be individuals with important legal rights at stake but who lacked expert legal help. If they were to participate in the study, we could observe the challenges that they were experiencing when completing Form 8, but we would be unable to assist, even in the face of observable errors that could compromise their legal rights. Although it might have been possible to build in subsequent legal help—for example, after the participants completed the study, they could be provided with a lawyer to help them redo the form—this would have exceeded the budget for this study. Additionally, the questions asked or the information revealed with third-party researchers present would presumably not be subject to solicitor–client privilege and, as such, could potentially be used prejudicially against a participant in later proceedings.

One important consequence of not using real litigants in our observational study is the inability to fully capture any impact of user emotional stress/distress. Family law litigation is notoriously stressful. Empirical studies have confirmed, unsurprisingly, that emotional stress can negatively impact cognitive performance.Footnote ⁴⁴ We would, therefore, expect that difficulties experienced by our study participants would be compounded in the experience of at least some real litigants, given the emotional intensity of the subject matter.

2. Results and Observations

In general, our observations from the think-aloud interviews confirmed the usefulness of a Guided Pathway approach for reducing user challenges when completing court forms.

2.1 Interface Advantages

There were several features of the Guided Pathway digital interface that improved usability.

First, the digital format allowed “just-in-time” information or contextual help through hyperlinked definitions and links through which users could learn more about a specific prompt. These features appeared to mitigate user difficulties with understanding legal and technical terms.Footnote ⁴⁵ We observed, for example, that participants who used the Fillable-PDF Form 8 often expressed frustration with the “legal jargon” used and the lack of definitions. In some cases, these participants did not express difficulty but did misunderstand the meaning of some of the legal terms that they engaged with: for example, multiple Fillable-PDF Form 8 participants selected “costs” as a claim but misinterpreted this claim as relating to expenses such as moving costs or financial support generally when it, in fact, specifically relates to legal costs associated with the proceeding.Footnote ⁴⁶ In contrast, participants using the Guided Pathway generally made repeated use of the hyperlinked definitions associated with claims when trying to decide what claims to choose. In their debrief interviews, several participants identified these definitions as being particularly helpful.

Second, the ability of the Guided Pathway to “streamline”—that is, only present to a user what they need to see for their particular situation—reduced opportunities for confusion.Footnote ⁴⁷ For example, one area that Fillable-PDF participants found confusing was a section with several blanks following a heading of “to the respondent” (see Figure 1).

Figure 1. Excerpt of Form 8.

Although a court clerk would typically fill in this section, participants who were assigned the Fillable-PDF version attempted to complete the section and expressed difficulty with the prompts, especially with respect to whether their case was on the “fast track” or “standard track.” In our 2016 quantitative study of Form 8, this section was identified as complex because, among other things, an applicant could easily be confused about whether to fill it out. By not requiring a user to view this material, the Guided Pathway eliminates the potential for this sort of confusion.

Similarly, Fillable-PDF participants also expressed difficulty in navigating the portion of the form in which they needed to choose their claim (Figure 2).

Figure 2. Excerpt of Form 8.

Multiple participants expressed confusion about which legislation and court applied to their claims. In the words of one participant: “I didn’t even understand what was going on with the different rows [sic].”Footnote ⁴⁸ Another participant stated: “[i]t doesn’t really explain what these terms mean […] or what the different acts are […] because I filled it out by myself I found some parts confusing.”Footnote ⁴⁹

Participants assigned the Guided Pathway did not experience the same challenges when navigating the list of potential claims. One reason for this was the different presentation of information—on the Guided Pathway, the claims were not organized in columns and with reference to different courts and legislation, but were instead presented in single lists, which were in turn curated depending on the initial description of the user’s situation (e.g., a user without children would not see the claims relating to child support).

Third, the Guided Pathway hyperlinked to relevant external resources, including government child support guidelines and table amounts. This type of feature mitigated a source of frustration: the Fillable-PDF version participants repeatedly expressed that they did not know what “table amount” meant in relation to child support and that they wanted more information.

1. The Value of a Functional Literacy Approach in the Design and Evaluation of Court Form Guided Pathways

The results of this study confirm, in our view, the value of using a functional literacy approach combined with robust human testing to identify challenges that users might experience when interacting with court form Guided Pathways.

First, as summarized in Part II, after applying a functional literacy framework to evaluate their Guided Pathways, CLEO concluded that this was a useful approach for locating potential trouble spots and guiding targeted solutions. This part of our study provides a “real world” demonstration of how a functional literacy framework can be used to help design and evaluate court form Guided Pathways.

Second, in our view, the usefulness of a functional literacy approach was confirmed in the “think-aloud” portion of our study (summarized in Part IV) in which the participants’ experiences were largely consistent with the results of our 2016 quantitative study of Form 8 when using the Evetts and Gauthier task complexity assessment system. Those parts of Form 8 that were identified as complex in the 2016 quantitative study when using a functional literacy framework also tended to be experienced as complex by participants who were using the Fillable-PDF Form 8 to conduct the think-aloud. This correspondence provides some measure of validation that those parts of a court form that the Evetts and Gauthier system will identify as difficult will, in fact, be experienced as difficult by users.

Although these observations were made in the context of studying one family law court form, they have broader resonance: they suggest that a functional literacy framework can be generally useful in the design of court form Guided Pathways to ensure they are optimally accessible to the public.

In our work, we first used the Evetts and Gauthier approach in 2016 to assess the complexity of paper or PDF-based court forms, and again used this approach to assess interactive Guided Pathways for the current study described in this article. We chose the Evetts and Gauthier system because of its systematic task-based approach. That said, there have been other approaches to assessing form usability that have been published subsequently to Evetts and Gauthier’s 2005 work that also adopt a functional literacy perspective and may be also usefully used to assess court forms. A recent example is Steenhuis et al.’s use of the framework set out by Jarrett and Gaffney in Forms that Work when developing a web application that can be used to measure and improve the usability of court forms.Footnote ⁶⁰

Regardless of the specific framework chosen, it is important to recognize that a functional literacy approach will not identify all challenges that users might experience with Guided Pathways. It is just one diagnostic tool that can be deployed alongside others. For example, persons with physical and mental disabilities may need particular measures in order to provide them with effective access. Additionally, in the case of online tools, not everyone has access to the hardware (e.g., computers or smartphones) or the appropriate level of “digital literacy” required to effectively use them. A functional literacy framework is not designed to identify all types of accessibility concerns.

Moreover, as discussed in Part I above, the functional literacy framework may miss certain barriers. This was demonstrated in our current study: although the outcomes of our think-aloud interviews largely confirmed our 2016 quantitative complexity analysis of the Fillable-PDF Form 8, they also revealed additional potential barriers that were not previously identified. Testing with human subjects can provide important additional insights.

However, robust human testing in relation to proposed access-to-justice measures can also come with ethical challenges, as observed above. One way to address some of these ethical issues but also hear from real users is to collect anonymized user data, as we did in the second part of the study.

The value in analyzing anonymized user data will be dependent, of course, on the user data that the system actually collects. As noted in Part III above, in the course of this study, we observed that we did not have access to all of the anonymized data that we might have wanted to study. At the outset of the project, CLEO believed that their system (the reuse of a publicly funded existing system) would collect many of these data points. However, missing data and relationships between different data sets were identified through the research. One important outcome from this project was that CLEO took steps to have their Guided Pathway platforms gather more data points from users.

Theoretically, some of these issues might have been avoided by taking an “evaluation by design” approach at the outset. This would have involved identifying, at the design stage, what data points the system should be capturing and how best to capture them, in order to assess access-to-justice impacts.Footnote ⁶¹ But, practically, designing to capture all the needed evaluation data upfront is not entirely straightforward or always possible. There are serious budgetary, time, and privacy and security implications involved with data capture that are beyond the scope of this article but create their own limitations. Taking an iterative approach to data collection within the confines of these practical boundaries, as CLEO has done, is good practice.

2. Online Interactive Court Forms as Access-to-Justice-Enhancing Tools

The results of our study strongly suggest that a Guided Pathway approach can yield significant accessibility benefits for many SRLs.Footnote ⁶² These benefits were confirmed in both the anonymized survey data and through the think-aloud sessions.

As discussed in more detail in Parts III and IV, the digital format of Guided Pathways proved to assist non-legally trained users by allowing additional explanations to be available “on demand” and helpfully streamlining the form (e.g., if only 20% of the form is relevant to a user, a Guided Pathway can show the user only those parts of the form and not confuse them by also showing the irrelevant 80%). Additionally, our study revealed that the Guided Pathway offered improvements that were largely independent of the digital format and resulted instead from a design approach that centred SRLs as users. These improvements included greater use of plain language and the addition of contextual information.

At the same time, our research underscores that design reforms cannot alleviate all the difficulties that are experienced by family law SRLs. First, the full meaning of some terminology may not be captured in “plain language” alternatives.Footnote ⁶³ In some cases, terminology may be simplified at its source by replacing technical terms with more common and simple language (e.g., replacing “respondent” with “your partner”), but this is not always appropriate. It would be a disservice, for example, to convert “arbitration” into a different plain-language term or phrase when the prompts relating to arbitration are referring to a very specific legal concept. As Rabeea Assy has observed in his scholarship on the limitations of the plain-language movement in the legal context, “technical expressions [in law] are typically incomprehensible to laypeople because their legal meaning is not contained or exhausted in their linguistic meaning; accordingly, the use of ordinary words would not dispel their unintelligibility. Their legal meaning can be grasped only by possessing specialized knowledge of the legal context in which these expressions operate.”Footnote ⁶⁴ While a Guided Pathway can substantially reduce terminological challenges by using simpler terms upfront or, alternatively, providing more information about the meaning of a term, it is unlikely to eliminate all terminological challenges for every user.

Second, our study reveals that some users will find the length of time that it takes to fill out the form taxing. However, the length of the process cannot necessarily be materially reduced. As one study participant acknowledged: “I see why [the length is] necessary because like anything dealing with court and law, they must like try to get enough information to make a fair judgement.”Footnote ⁶⁵ This participant’s comments echo an observation from Shannon Salter and Darin Thompson: “[d]esigning justice processes to meet the needs of users is not a matter of devising the simplest processes possible. There will naturally be a certain degree of complexity within a system that must resolve potentially complicated disputes.”Footnote ⁶⁶ Perhaps the best that can be done here is what CLEO does for the Form 8 Guided Pathway: set expectations for users at the outset about how long the process may take and provide appropriate context about how the different tasks that a user is prompted to complete are related to the interests that the user wants to advance.

Third, another area of observed continuing complexity was responding to open-ended prompts. A Guided Pathway format has some significant advantages here, as it can provide users with more explanation about what can or should be included in a response; prompts can also be broken down into subquestions that more directly elicit possibly legally relevant information. Notwithstanding these advantages, this was still an area that created difficulty for some users and participants. This is not surprising: because the information best included in response to these prompts is often highly personalized, it is difficult for a general system to provide contextualized, detailed, and prescriptive guidance to individual users that will always clearly orient them towards an ideal personalized response. Challenges in answering open-ended prompts are also presumably driven by the underlying complexity involved in crafting legal narratives. As Toy-Cronin et al. observe:

By adding subquestions, the Guided Pathway assists SRLs in crafting their legal narratives, directing them towards providing legally relevant information. The Pathway, based on the advice of practising family law lawyers, includes prompts that are based not only on the original court form, but also on the legal test set out in legislation and its interpretation by courts in previous cases. But even this form of guidance has limits. In a particular case, it is possible that a lawyer would see different information as being relevant than that elicited by the Pathway or suggest different ways of presenting the same information. As Branting and McLeod observe, “[i]t is the role of a legal counsel to identify the goals that the client hopes to achieve through a legal process and then to elicit a coherent narrative that is relevant to one or more possible legal remedies that could achieve those goals.”Footnote ⁶⁸ While the Guided Pathway provides some indirect access to legal expertise via subquestions that seek to elicit legally relevant facts, a residual challenge arising from the nature of legal reasoning is likely to remain.

In general, the issue of how to successfully elicit legally salient and effective narratives is a challenge for those seeking to design A2J-enhancing digital tools for use without direct assistance from lawyers. As an alternative to the approach taken by the Guided Pathways (i.e., adding subquestions in an automated form), Branting and McLeod have, in recent work, explored a more dynamic “narrative-driven case fact elicitation.”Footnote ⁶⁹ Following the public release of ChatGPT in November 2022, there has also been growing interest in whether generative artificial intelligence (AI) can provide a more “conversational” means by which the public could interact with digital tools to describe their factual circumstances and craft those into the narrative that is most effective for addressing their legal needs.

3. Final Thoughts

The nature of family law and other areas of “personal plight” law may mean that having expert legal assistance is ideal. But the reality is that the vast majority of people cannot access legal assistance. We need immediate and practical solutions to assist SRLs.

We know that conventional court forms are generally quite challenging, if not virtually impossible, for SRLs to complete. The results of this study point towards one type of measure that can significantly reduce complexity: the use of court form Guided Pathways, particularly when these Pathways are designed with literacy principles in mind.

In detailing the results from our multifaceted case study, we hope that others who are interested and engaged in court form design can gain insights into the specific strengths of the Guided Pathway approach, and how to optimally design and evaluate the usability of court forms offered to the public through this type of digital format.