Understanding The Scripted Writer Score

Scripted developed a new writer score to improve content quality and reward our best freelancers.



Earlier this fall, we announced a major change to the way Scripted writers were evaluated. For years, our writers' ability to claim work was based on their various Industry scores; now, this has been replaced by a single writer score based on our editors' and customers' reviews of written content.

So what did we change -- and why did we do it?

The Flaws With The Industry Scoring System

The Industry scoring system only had two main inputs: writers' scores on their original Industry applications, and how punctual they were from that point on. Given all of our writer applications are peer-reviewed, too much of a writer's destiny on Scripted was based on the score received during that initial review process.

From that point on, the only thing a writer could do to get access to more work was continue to turn things in on time -- with no specific rewards for jobs done well.

Getting To An Actual Measurement Of Quality

So what could we do to make sure our writers were getting rewarded for their knowledge, effort, and diligence? We decided to use the best data we had at our disposal: reviews of content from our editors and customers.

First, we reworked the questions that our editors are asked when reviewing jobs, ensuring that those closely matched the areas most critical in determining whether a job met -- and surpassed -- expectations. We then spent several months reviewing and analyzing this data, experimenting with different algorithms to structure that data to give us a writer score that matched historical performance trends and customer preferences.

Thanks to our skilled Data Science team, we were able to boil this down to a single writer score that truly measures what's important to our customers.

The Components Of The Writer Score

The new Scripted writer score consists of four components: quality, originality, research, and guidelines.

• Quality is an evaluation of how well the writer crafts sentences and paragraphs. This includes a measurement of spelling and grammar as well as ease of comprehension.


• Originality focuses on a writer's ability to be creative and avoid repetition. Writers are rewarded for varying sentence structure and language.


• Research highlights how effectively a writer incorporates facts and insights. Depending on the type of content, this can be a blend of the writer's previous relevant experience and research done specifically for the post.


• Guidelines measures the writer's follow-through on a customer's specific requests on formatting, structure, and tone. Writers can do well in this category by ensuring they pay close attention to all of a customer's keyword requirements and special instructions.

These four categories are weighted to produce a single writer score that influences a Scripted writer's ability to get access to work.

Unifying The Scripted Experience Around The Writer Score

Now that we have our secret sauce, we're incorporating it throughout the Scripted experience for writers and customers.

When writers apply to Scripted using our revamped writer application, they are evaluated on the same four categories as they will be once they start doing work for Scripted. This ensures that our application gives writers a representative experience of what it will be like to interact with Scripted customers once they are approved.

Each job that an approved Scripted writer pens is graded soon thereafter on quality, originality, research, and guidelines. This feedback is incorporated in real time into the writer's score, ensuring that the system's measurement of writer quality is constantly up-to-date.

In the near future, we'll be rolling out a writer feedback tool that will allow our writers to see how well they are doing in each of the four categories. The Scripted writer dashboard will give writers an understanding of which areas they excel in and where they might be able to improve in order to gain access to more work.

Finally, we'll also start surfacing this information to customers. When browsing topics in the Scripted Topic Marketplace, customers will soon be able to see badges on writer profiles that identify our top performers in each of the categories. This will help customers better understand a writer's strengths when making purchasing decisions.

In Search Of Constant Improvement

One thing differentiates Scripted from all other content marketplaces: rather than simply providing a listing of writers and rating them as five-star or one-star performers, we strive to work with our freelancers to help them become more productive and to match them with customers whose preferences match their strengths. With the revamped writer score and all the associated changes, we're confident that our writers will have more tools than ever before to continue this process of improvement.

However, given what we ask of our writers, it's only fair that we also look inwards and constantly ask ourselves how we can do better, too. The new writer score is a step in the right direction, but we need to keep working to improve our product and our community -- and we'll always be grateful for your feedback as we continue on that journey.

---

Published by Karan Lodha on Thursday, December 3, 2015 in Writing.