The Big Data Tech behind Times Internet’s Native Ad Play - Colombia

Not many people may know it, but one of the largest publishers in India, The Times of India Group, is also home to one of the largest publisher-owned ad network platforms in APAC. The Group’s digital venture, Times Internet Ltd (TIL), runs one of the most complex and sophisticated ad serving operations, in addition to hosting the editorial content for multiple sites (internal as well as partnerships).

Around a year back, TIL had launched its own native ad platform, called Colombia. (Native ads are personalized ads that are shown to the web users based on their past browsing history, interests, etc., and are usually text ads as opposed to display/banner ads.)

Given the growing global trend of more and more brands putting their money on native ads, platforms such as Colombia are gaining increasing significance in the media world. The platform ensures similar user experience across mobile and web.

I recently spoke to Sumit Malhotra, Head – IT, TIL, to know more about how they developed Colombia, the technology behind the platform and challenges associated with what Sumit calls a “big data recommendation system.”

It would be pertinent to note in this context how the TIL ad network has grown in the past two years, which is nothing short of exponential—from serving around 40 million ad impressions per day in last June to a peak of 500 million impressions/day this month.

Besides serving ads from its own marquee properties, TIL has tie-ups with third party ad networks like Taboola and many others. So the ad inventory that is served through Colombia comes from a number of sources, all of which need to be integrated tightly with the TIL platform for serving to the user—who could be sitting and surfing in any part of the world. 

The key, says Malhotra, is to provide a consistent experience to the audience with as low latency as possible (often, the native ads are served in 100 to 150 milliseconds but, in any case, the threshold has to be kept below 500 ms).

“Otherwise, the user will have either scrolled down the page or moved on to another site (without seeing or clicking on the ad),” he says.

The biggest challenge for any ad network today is to deliver an ad at a low latency. To do so, it is important that calculations and permutations related to which ad is to be served to which user profile are based on the recommendation of the big data system—Colombia in TIL’s case—which is run in-memory rather than on disk.

Talking about the challenges, Malhotra says, “Another challenge is that suppose a user is coming from the US and hitting our servers in India, so the travel time for a data packet is quite high; we need to take it closer to the audience. And since every ad is personalized, that is a big challenge.”

Part of the low-latency challenge is solved by having multiple ad server clusters in different geographic regions of the world. TIL has its servers hosted in different geographies and uses a mix of public cloud options and its own data centers. “This helps us serve ad requests within specific regions,” he says.

Another very notable and significant thing is that TIL has custom-built its own big data engine using open source tools and technologies—all done in-house by its 100-plus technology development team.

It took slightly over a year to build Colombia and then another 9 to 12 months to roll it out fully across the board for all online properties.

“Colombia was launched about eight months to a year back. And given that the infrastructure is huge, we had to roll it out to all the properties, 40 different brands across Times Internet. So the deployment also took time, as the technology needed to be integrated with the publishers as well. And after rolling out, you understand the issues and then scale it up slowly,” says Malhotra. Now it is fully deployed not only across all TIL properties but with the third parties as well.

The key benefit is derived from “the complex algorithms that help us run highly targeted campaigns,” he says.

Native ad platforms are one of the primary reasons why today’s users get the sense that if they go to particular kinds of sites (sport, technology, housing, etc.), the ads that accompany on the sites that they visit next are mostly related to the content they had just viewed.

“Given that it’s a personalized ad network, we need to do in-memory data recommendations. For maintaining low latency, we cannot afford to do any calculations on data that goes to the disk, all calculations have to be done in-memory,” says Malhotra.

Malhotra says that most components of big data analytics systems today run on bare metal servers rather than virtualized ones, as the virtualization adds another layer to the process and increases the latency. With the scale and complexity that TIL operates in, he says, “we cannot survive any virtualization overhead.”

On being asked why TIL didn’t go for a branded big data analytics platform (from known large vendors), he had this to say: “Because those vendors cannot give you that kind of personalization in-memory at this rate and at this kind of a size/scale. So for example, if we are using, say, 300 servers for our open source solution, it would require 600 or more servers to do the same things on vendor products.”

Also, Malhotra is of the opinion that if one uses a vendor product, one gets locked on to it and it also takes away the flexibility.

So it looks like TIL is not going to take the outsourcing route in the near future.

To keep its ad network in good health, TIL also has to do a lot of monitoring of how the ads are being served across different geographies. One, to check if a consistent experience is being delivered to the user; and two, whether a campaign is not surreptitiously trying to deliver some malware to the end user’s device (in which case the campaign is stopped and/or blocked.)

For this, TIL uses a mix of in-house resources as well as tools from third-party providers.

The challenge now is that, sometimes, the traffic at multiple sites can peak at the same time, which requires a different kind of scalability.

(Note: This blog post first appeared on dynamicCIO.com)

Why the New Coke Ad is More Sad Than Happy

I also like the ad that shows happy kids singing, but there are unhappy truths behind fizzy drinks people should know as well

Before I write any bad things about Coke, Pepsi or any other carbonated beverage company, let me first make an admission. I have had plenty of these drinks in my life and enjoyed them as much as kids still do today.

But then, I was also ignorant like most kids (and though I'm still ignorant to a big extent, I've also learnt a few things along the way).

So, what has changed?

For one, I started drinking fewer fuzzy drinks and began reading more about health issues, water problems and environmental destruction.

But more importantly, I became a father twice over – and began to get worried not only about what I consume but what I allow my kids to have. And when I see my young kids crave for a Pepsi, Fanta or another carbonated drink, and when I relent and let my wife share some with them and see them get aggressive or irritable after that...that's when I realize that a lot has changed, must change, from how things are now.

And that's why when I saw the new Coca-Cola ad in which kids sing a paean of hope while joyous stats are displayed on TV screens, I felt that this is wrong, just so wrong.

It's a great ad and I really like it – I only wish it weren't promoting fizzy drinks in the name of happiness and hope. Not to say it comes from a company that allegedly has double standards and dubious practices in the name of an iconic brand (to throw some light on this, readers may point their browsers to this interview of Michael Blanding, who wrote the book The Coke Machine: The Dirty Truth Behind the World's Favorite Soft Drink. And if you search “Truth + Coca-Cola” on the Web, you'll be swamped with millions of links, many of them quite revealing).

But unlike me, there are millions of people, many of them impressionable children, who will watch this ad and find happiness in bottles of Coke rather than more health-giving fruits, milk or even water. Few would work for the causes espoused in the Coca-Cola ad; most would just sip a drink and be happier than before. Or at least live in an illusion of happiness. Like I had been doing, for quite a while, unfortunately.

My first eye-opener about our current state of food-and-beverage industry came a few years back when I read Fast Food Nation by Eric Schlosser. In this book, Schlosser went deep into the working of McDonald's and revealed the smudges behind the golden arches – all in their gory details (the conditions and manner under which cows are killed and processed at a factory that supplies beef, for instance).

But I wouldn't like to single out Coke, Pepsi or McDonald's for the current sorry state of public health – which is increasingly getting riddled with obesity, hypertension, diabetes and scores of other ills. It has much to do with the entire process of looking at what we eat and drink from the profiteering lenses of giant corporations. Corps that are more concerned about economies of scale and the money they make rather than what ingredients are used and what effects the products have on people's health.

Kids in India (as probably in most places before the assembly-lining of food and beverages) used to enjoy home-cooked snacks and occasional indulgences of treacle toffees or aampapad (a locally made sweet of natural mango). Now they are gorging everywhere and all the time on industrially made 'addictive' wafers, fat-inducing burgers and large doses of carbonated sugar water. These and many similar obnoxious things is what you see all around you – in malls, at the local grocery shops, even at remote hill stations.

So dads like me (and moms, who aren't like me at all :) are forced to buy unhealthy things for their children if they ever want to step out of the house.

Some of us are trying to control and minimize the damage – but our little blogs and tweets get drowned in three-billion-dollars-a-year worth of massive advertising unleashed by the likes of Coke. And our tips and messages and efforts are lost in the sheer availability of “junk stuff” and the irresistible pull of the “convenience factor.”

Will someone please make an ad on that and highlight the real thing?

Let's celebrate and be joyous in the new year, but let's do it for the right reasons and for the real heroes – not for some cunning company.