SocialCrawlytics: Providing Insight On How Your Content Is Being Shared

It’s funny how subjective using social media tools can be from person to person. Every day, I use at least a half dozen tools to post and track social activity online, primarily SocialFlow, Chartbeat, Google Analytics, Tweetdeck, Facebook Insights, Exact Target.

On occasion, that toolset grows: Google Webmaster Tools, Simply Measured, PageLever, Followerwonk, Raven Tools, Curalate, Twitter Analytics, bitly, Muckrack, Tweetreach, Google Insights, Statigram, IFTTT… and there are more if I count the tools I use on a personal basis. I haven’t used Union metrics for Tumblr yet (frankly I am sitting on my hands until something cheaper comes out), and just today our Founding Editor Danny Sullivan introduced me to another potential tool, Crowdbooster (see this week’s Link Week column from Julie Joyce for more about that).

Many of these tools listen, track performance, provide competitive data, even make recommendations. They are all unique – although they may be working with the same datasets, or use the same APIs, they each provide something useful. Yet despite all of this data and opportunity, aggregating and understanding how our content is being shared across the most popular social channels simply just seems daunting.

Fortunately, while during our end-of-the-year posts, I had the opportunity to dig into Social Crawlytics, a social analysis tool that crawls sites and generates a report that shows how often each URL was shared on different social platforms.

There’s no easy way to run a report like this in Google Analytics (that I know of anyway), and looking at individual URL metrics within social platforms is not efficient. I had the chance to talk to Yousaf Sekander from RocketMill, who created the tool, and asked him a few questions on how the tool works.

Getting Started

Social Crawlytics is free when you first start using it. Once you sign up, you’ll notice you’ll have 5,000 credits. Each credit is for one page on a domain, so you’ll be able to crawl and analyze 5,000 pages right away. You can donate to get more pages, or you may consider signing up for the enterprise level program, depending on what’s best for you.

Be warned: it can take anywhere between 1-6 hours to crawl 5,000 pages, so be prepared to build in some time to generate your reports.

Scheduling Reports

When you are setting up the domain you want to crawl, you have the option to schedule a report. You can choose to skip this step, but if you prefer to monitor a domain regularly, scheduling a report is a good option.

Yousef recommended that a sites like Search Engine Land or Marketing Land that generate a lot daily content and social activity should be scheduled at least twice a month in order to get a historical snapshot of shares for each URL. Of course, each time a you run a report for a site with thousands of pages you’ll use up your credits rather quickly, so use your best judgement.


The last step in generating a report is simply filling out the notification form, this will let you know when the crawl is complete and the data is ready to be viewed. This is important: you don’t want to download data that’s incomplete.

Social Shares Reports

Under “Reports” in the left navigation you will see the domain for reports generated – the most recent report will be listed at the bottom. Yousef mentioned that he is still tweaking with the reports, and plans on adding dates to these soon. If you are regularly monitoring a domain and have a report scheduled, it creates a new report each time the crawler runs. All metrics are cumulative, so the most recent report will show the total shares generated at that time.

Below is the last report we ran for Marketing Land, you can see right away that our posts are monthly shared on Twitter, Followed by LinkedIn.

You can also see the most popular authors based on social stats alone. the colored graph are the Page Shares Per Network, where Social Crawlytics takes the top 50 shared pages, and brakes them down by network. Each color block represents a URL; for example, a recent post called Content Marketing for People Who Hate Writing took off on StumbleUpon. That same shade of yellow is represents the same URL in the rest of the graph.

One of the most useful aspects of the tool, however, is the .csv export. Although you can see page-level results within the dashboard, exporting the data as a spreadsheet allows you to customize results and sort by the number of shares that occurred on a specific social channel: Facebook, Twitter, LinkedIn, Digg, Delicious, StumbleUpon, Google+ and Pinterest.

You can also sort by the total number of shares a specific URL received. The tool also goes one step further by including the anchor text pointing to the URL, and the header status code for that page.

Final Thoughts

I asked Yousef his motivation behind building such a great tool:

“Creating compelling content is not cheap nor is it easy, it is imperative to carry out research to find out what type of content has worked for your competitors. I created Social Crawlytics because there was nothing out there that could allow me to carry out this type of research, particularly for large websites.”

Yousef also repeatedly mentioned that Social Crawlytics is a free service, and he does not have any commercial motives at this time. “It is a tool made out of love and passion,”  he said, “I am footing the server bills myself… I don’t ask for money but if a user thinks the service is worthy or useful they can choose to donate.”

Personally, with the countless tools I use regularly,  I can attest that this tool fulfills a professional need, not just for social media marketers and community managers, but for any business who markets online and relies on site traffic and visibility. And not just for internal business reasons, but for competitive analyses as well.

How will you use a tool like this? Share your ideas in the comments below.

Related Topics: Channel: Analytics | Features & Analysis | Marketing Tools | Marketing Tools: Analytics | Marketing Tools: Social Media | Social Media Marketing | Top News | Uncategorized


About The Author: Monica Wright serves as Director of Audience Engagement for Search Engine Land and Marketing Land, two of the leading trade publications for the digital marketing industry. With over 15 years of experience in online publishing, content marketing and audience development for media companies, she is focused on content consumption and measuring user engagement on both sites across multiple platforms including desktop, mobile, social and email. In addition, she serves as a program coordinator for the SMX conference series, produced by parent company Third Door Media, publisher of Marketing Land and Search Engine Land. You can find Monica on Twitter, LinkedIn and Google+.

Connect with the author via: Email | Twitter | Google+ | LinkedIn

Marketing Day:

Get the top marketing stories daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. You can read more about our comments policy here.
  • Maks

    Nice…. – download music, free

  • Maks

    Nice… – download music, free

  • Eva Diz

    Crowdbooster is a great tool, i’ve been working with it ​​for months and is among the best. I dont know Social Crawlytics, but it will be the next to try. Thanks for sharing such good information.

  • Coffy Group

    I don’t think I’ve had the opportunity to explore such tool to measure data for my social media efforts, but you’ve listed great information Monica. Reading through the many advantages this tool can offer only makes me realize how much I’m missing. I’ve been doing social media for sometime now and the data collected are rich and should be fully maximized. I can’t think of a reason (at the moment, anyway) why I didn’t find a tool such as Social Crawytics before, but I’m set to correct that very soon. Thanks for the read.

  • Monica Wright

    I heard about it this fall, but didn’t really use it until December when I wanted to see what our most-shared posts were for the year. You’ll like it.

  • Monica Wright

    I’ll be digging into CrowdBooster this week! Thanks!

Get Our News, Everywhere!

Daily Email:

Follow Marketing Land on Twitter @marketingland Like Marketing Land on Facebook Follow Marketing Land on Google+ Subscribe to Our Feed! Join our LinkedIn Group Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Marketing News!

Marketing Day is a once-per-day newsletter update - sign up below and get the news delivered to you!