In the context of data labeling and annotation projects, effective task management and quality control are crucial for ensuring high-quality results. One key aspect of this process is the assignment of specific roles, such as annotators and reviewers, to individual tasks. Labelo provides robust features to facilitate the assignment of these roles, helping streamline the workflow and maintain consistency across the project.

Who are Annotators and Reviewers?

Annotators

Annotators are responsible for the initial labeling and annotation of data. They apply labels, draw regions, highlight text, and perform other necessary tasks to categorize and annotate the dataset according to project guidelines. Their work forms the foundation of the labeled dataset, making accuracy and attention to detail essential.

Reviewers

Reviewers oversee the quality and consistency of the annotations performed by annotators. They review the annotated data to ensure it meets the required standards and adheres to the project’s guidelines. They provide feedback, request revisions, and approve or reject annotations, playing a critical role in maintaining the overall quality of the dataset.

Assigning Annotators

Assigning annotators is a crucial step in managing data annotation projects efficiently. By assigning tasks to designated annotators, you ensure that each task is handled by trained individuals, improving the accuracy and consistency of the annotations.

To begin with, navigate to the annotation settings within the project settings, as depicted in the figure above. This section allows you to configure how tasks are assigned to annotators.

It’s important to note that annotators can only be assigned to tasks if the task distribution type is set to manual. This setting ensures that you have direct control over which annotators are assigned to specific tasks, allowing for more precise management and oversight of the annotation process.

To enable manual task assignment, you need to adjust the task distribution settings of the corresponding project. Within the annotation settings, look for the option to set the task distribution type. Change this setting to manual, as shown in the figure below.

After changing the task distribution of the project, navigate back to the tasks page.

Select the task to which you want to assign annotators. This will trigger a dropdown menu where you will see the option “Assign Annotators,” as shown in the figure above since we have set the task distribution to manual.

Here initially the ‘Assign’  button will be in the disabled state, once we move the members under ‘To Be Assigned’, then they will be highlighted.

When we choose the option to assign annotators, a modal shown below will appear. The modal contains all the annotators from the corresponding project members. From there you can choose the annotators to be assigned for the specific task and click on the right arrow button so that they will be moved under to be assigned members. After that, we can assign those selected members to annotate that task. 

Along with annotators, Labelo provides access for the organization’s owner and administrator to annotate the task.

Besides these, suppose you want to unassign any member from annotating the task, you can just select the particular member from ‘to be assigned’ and click on the left arrow button so that they will be moved under ‘members’, and they will be unassigned to the task.

Labelo also provides a feature to search for the members you want, which allows users to find the members easily.

By following these steps, you can efficiently manage the assignment and un-assignment of annotators, ensuring that the appropriate team members handle each task. This structured approach helps maintain an organized and effective workflow, ultimately contributing to the success of your annotation project.

Once we assign an annotator, they will be shown in the annotator’s column shown above as faded since they are only assigned but not labeled the task. Soon after they labeled the task, it will turn into highlighted, as in the figure below.

Key Features:

• 1. During annotation, only annotators, owner, and administrator are shown in the modal. No reviewers are shown there.
• 2. You can assign annotators for multiple tasks at a time, but cannot un-assign them at a time.
• 3. When a member has already labeled a task, then you cannot assign him again for the same task.

Assigning Reviewers

Assigning reviewers is a critical step in the data annotation process, ensuring the accuracy and quality of the labeled data. Reviewers play a vital role in overseeing the work of annotators, verifying that annotations meet project standards, and providing feedback for improvement. 

Similar to assigning annotators, first you need to navigate to the review settings within the project settings and change the task distribution. With this, the option for “Assign Reviewers” will appear in the actions dropdown.

Here you will have the option to ‘Assign Reviewer’.

Within the modal by choosing the “Assign Reviewers” option, you can see all the reviewers as well as the owner and administrator of the corresponding project listed. From there you can choose the reviewer you want to assign for the particular task.

As we saw in the assign annotator feature, you can unassign the members from reviewing by clicking on the left arrow button.

The assign Reviewers model also includes a feature to search for members.

Similar to what we have seen in the assign annotator feature, the assigned reviewer will be shown in the reviewer’s column as seen in the image below. 

Once the review is completed, and if the annotation is accepted, it will be highlighted along with a checkmark icon. Otherwise, if the annotation is rejected, a cross-mark icon will be displayed.

Here it is clear that the annotation is accepted by the reviewer since there is a checkmark icon along with the user pic.

Here we can see the annotation is rejected by the reviewer.

Key Features:

• 1. During reviewing, only reviewers, owners, and administrators can review the task. Annotators will not have access to that.
• 2. You can assign reviewers for multiple tasks at a time, but cannot unassign them at a time.
• 3. When a member has already reviewed a task, then you cannot assign them again for the same task.

In conclusion, Labelo’s robust role assignment features empower teams to efficiently manage tasks and uphold quality standards in data labeling and annotation projects. By clearly defining roles like annotators and reviewers, Labelo ensures a streamlined workflow, enhances collaboration, and fosters consistency, ultimately contributing to the delivery of high-quality, reliable annotations.