Twitter Cuts Off DataSift To Step Up Its Own Big Data Business

In the push for more revenue growth, Twitter has been building up its business in areas like advertising and commerce, but a move made late Friday night points to another area where the company is setting its sights: big data analytics.

Twitter announced that it will be terminating agreements with third parties for reselling firehose data — the unfiltered, full stream of Tweets and all related metadata that goes along with them.

Instead, it will use its own in-house big data analytics team, which it developed around its acquisition of Gnip in 2014, to seek to build direct relationships with the data companies, brands and others that use Twitter data to measure consumer sentiment, market trends and other moving targets that can be better understood by tracking online conversations — a transition it says it hopes to have completed by mid-August.

DataSift, the biggest company to be affected by Twitter’s move, services thousands of businesses who in turn serve thousands more. Unsurprisingly it moved quickly to post its own reaction to the termination and its own determination to push ahead in its own business.

NTT Data, which deals only in Japanese Tweets, is still listed as a Twitter firehose partner at the time of writing, but Twitter has confirmed to me that NTT is also affected by Friday’s announcement.

This is both a very unsurprising and sudden move, from the looks of it.

Talking to Nick Halstead, the CEO and founder of DataSift, he said that his company was “blindsided” by Twitter’s announcement, which it made without any warning to DataSift. He said that before this, the pair had been discussing a renewal of the deal. And while DataSift recently added Facebook — Twitter’s big social advertising competitor — as a firehose partner, it didn’t appear that this would impact those discussions.

“We were in the middle of negotiations with everything pointing to Twitter wanting to still continue to be a part of an open ecosystem,” he said, “but this is clearly now not true.” It also comes at a somewhat awkward time for the startup: it’s currently in the process of raising a new round of funding — a Series D round. To date, DataSift has raised nearly $78 million.

On the other hand, for those who have been following how Twitter has grown as a business, the company’s move to cut off third-party firehose relationships should come as no surprise.

The company has made no secret of its bigger philosophy about how it interfaces with third parties in general. In its (in)famous ‘quadrant’ diagram, the company outlined its position towards third parties that added value to what Twitter was doing versus those that effectively overlapped with Twitter’s own efforts: those who were building Twitter clients that “mimic” Twitter’s own experience in reproducing the Twitter stream were getting cut off.

Last year’s move by Twitter to acquire Gnip — another firehose reseller who competed with DataSift — was seen at the time mainly as a response to Apple’s acquisition of Topsy, who had been another firehose partner. DataSift went so far as to reassure people that its status with Twitter would not be affected.

But now it’s clear that Twitter had other things in mind, too. Zach Hofter-Shall, head of Twitter ecosystem, said as much in his blog post late Friday:

“One of the reasons Twitter acquired Gnip was because Twitter believes the best way to support the distribution of Twitter data is to have direct data relationships with its data customers – the companies building analytic solutions using Twitter’s data and platform,” he wrote. “Direct relationships help Twitter develop a deeper understanding of customer needs, get direct feedback for the product roadmap, and work more closely with data customers to enable the best possible solutions for the brands that rely on Twitter data to make better decisions…The acquisition of Gnip was the first step toward developing more direct relationships with data customers.”

In fact, whether they wanted to believe it or not, these companies were told by Twitter that they would be getting cut off nearly a year ago, we understand.

The direct relationships Twitter has with data customers, meanwhile, are also starting to take a new kind of form: just last month, Twitter made its first investment in a startup through its new investment vehicle. The recipient? Dataminr, one of the companies that analyses Twitter firehose data, in its case to track news and financial data.

The reason why Twitter wants to tap into more big data business, of course, comes down to one big reason: money.

Since going public, Twitter has regularly faced questions about user growth. On one hand, that has led it to many iterations as it tries to snag more consumers who are not already regular Twitter users; but it also has increasingly focused on ways that it can better monetise what it already has on its platform.

That’s where the big-data services come in. Twitter, from what we understand, was making very little revenue from the relationship it had in place with DataSift, with the latter only giving Twitter a 20% cut on any revenues it made from when it sold on Twitter data to its customers. Cutting off the firehose to DataSift, Twitter hopes, will potentially give it access to the customer deals that DataSift held before.

(The big question now will be whether Twitter manages to convince enough of the people who used to buy data through DataSift to turn directly to Twitter for those needs instead.)

“Twitter believes that creating a closed market for their data allows them to generate more revenue,” Halstead told TechCrunch. “We believe and others believe that an open ecosystem is important for a brand to understand what is going on in the market.”

As for where DataSift is turning next, the company says it is signing on more social networks to provide its own firehose data feeds — no comment from DataSift on which feed will be next but I’ll point out that it’s notable LinkedIn is not yet a partner but is clearly looking for more ways of using its data for analytics, too.

DataSift is also still able to work with Twitter data: if a third party buys data from Twitter, it can supply it to DataSift by way of a “connector” so that it can still be parsed by DataSift’s algorithms. However, this will mean significantly lower revenues for DataSift in the process from that feed.