Relying on external data feeds can be a pain and extremely annoying

[Peter Simpson]

But they are a necessary evil.

I have 2 different types of apps in the Play Store that rely on two completely different suppliers for data feeds, one is a weather app and the other is a service app. The one thing that really annoys me is that these websites often get DDoS attacks. I've managed to code around one suppliers website getting attacked and not supplying the XML feed, but only after my apps rating took a bettering. I now have another problem where my weather app took a feedback battering because www.worldweatheronline.com keep getting DDoS attacks and it can lasts for days. Well users of my app which cost $0.99 then leave one or two stars for the app, when the app was working 100% perfect those same user never left four or five star feedback, *^!*-"%$. Anyway I've now coded around not receiving the JSON feed in my second app due to DDoS attacks on my other suppliers website.

It just annoys me that idiots around the world seem to think that it's clever taking down websites for fun, it just disrupts everything for thousands of innocent people around the world.

So do you have any annoying hates that are 100% not in your control but you need and that can effect what you are doing?

 

[KMatle]

The website's owner(s) must take action about this (though you will not have any claim to get service from that site).

 

[Brian Robinson]

HaHa... funny I just wrote a similar complaint... I have around 400 users for my app... 6 people have left feedback... Yet one person finds a bug and he is straight onto the feedback..

It is just the anonymity of it I guess... Who would have the guts in this world to talk t someone in their face and say... "I know you gave this to me for free... But it is shit... And you have wasted my time by letting me use it"

Not many I say!

 

[Peter Simpson]

It's really annoying @Brian Robinson, especially as in my case the data feeds are 100% out of my control. I actually used to use a 3rd data feed but luckily for me a found a nice library on this community that catered for my needs, so lately I have removed an XML feed from one of my apps and I now use a library in it's place that has the formulas already built in, thus no internet needed

 

[sorex]

not really suitable for a weather app but if you work with other kind of datasets I prefer a scheduled parser that grabs and dumps the data to my own server so that I have a cached resultset.

but as said it's not suitable for realtime or (some) parameter depending fetching of data.

 

[Peter Simpson]

Weather feeds are usually updated by the feed suppliers every 15 to 60 minutes depending on the supplier @sorex. My weather app only checks every 90 minutes for a new weather feed and already stores the previous feed on the device. Your argument is irreverent as if you can't get the feed in the first place because of DDoS attacks then you just can't get the data feed, it's as simple as that, scheduled or not. http://www.worldweatheronline.com/ who are a major player in the industry were down for about 3 days not long ago, even their website was running seriously slow (if at all). When you can't reach a server for three days and customers are seeing the wrong weather report for example, the app said it's sunny because of the cache but in fact it's been raining for 2 days solid according to a customer, then everything just becomes a joke. Actually my cache is only available until the device is rebooted then it has to get new data.

BTW http://openweathermap.org was also down as that's the backup feed in my app. I was hoping to switch the seeds and upload the app to the Play Store, but that didn't happen either

I've also had the same issue with my other app which is a service provider, but they they now done something about DDoS attacks and I've coded around their issues.

 

[Peter Simpson]

I was sent this email, but their solution still didn't for for a good few days. Thinking about it, I reckon that their feed was down for about a week.

Dear API User,

As you all are aware that we have been under constant DDOS attack on our network for last few days. We have tried everything to get our solution back but it seems the attack is not subsiding.

Therefore we request all our API users to please change your request url from http://api.worldweatheronline.com to http://api2.worldweatheronline.com

Thank you,

World Weather Online Team

 

[Troberg]

For a weather app, it seems like a good idea to have several sites to pull data from, so that, if one goes down, you just try the next.

Or, the last fallback solution: Show a message box with the following text: "The weather service this app is using can't be reached at the moment. Hold the phone against a window, then look slightly to the left of it for the current weather status.".

 

[Peter Simpson]

 

[sorex]

Your argument is irreverent as if you can't get the feed in the first place because of DDoS attacks then you just can't get the data feed, it's as simple as that, scheduled or not.

that's why I wrote it's not suitable for weather apps as it requires almost live data to reflect the actual conditions.

for a news update of some site it won't hurt showing those from a few hours or the day before.

@Troberg 's solution is an excellent backup solution and fail/fool proof

 

[Peter Simpson]

@Troberg solution is a good idea @sorex but as I mentioned previously, I do have a backup supplier built into the app, but they were under attack too

After going a a free weather feed forum, it appeared that a number of feed suppliers were under attack and it lasted for a few days.

You are 100% correct though, with news feeds you can easily wait a few hours before updating the news feed . But then again if the news feed is down for a few days then that's bad news to (bad news, get it lol ).

At the end of the day, all DDoS attacks are just a pain in the rear end.

 

[sorex]

must be some people who hate all these weather apps, I'm surprised I'm not involved

most firewalls have ddos "protections". I wonder how that can even work when it can only react when the packets alread arrived thus the saturation is there already.
not reaching the webserver but if the line is saturated it will be unreachable aswell.

 

[Troberg]

DOS attacks are usually based on the principle of sending a tiny, broken package. The server responds with a "Hey, that was broken, resend!" package, which requires much more work and bandwidth for the server. Usually, the sender adress is spoofed, so the response don't even go back to the attacker. That way, an attacker with a fairly weak connection can still cause disproportionally large damage to a much stronger target.

On top of that, we have the DDOS attacks, which is the same, just that many people do it at once, increasing the effectivity.

One way to defend against that is to recognize that you are being swamped by bad packages and stop responding.

That said, much simpler "attacks" has been quite successful. When the Swedish search engine The Pirate Bay was attacked by 50 armed policemen, someone (not me, I promise) took down the police web site simply by finding a bunch of large images on it, creating an auto-updating html page with those images and spreading that page to many people. I wouldn't even call that an attack, as they were basically using the web site services as they were intended to be used. I find that more akin to, say, protesting against a shop by getting lots of people who don't intend to buy something to enter the shop and just wander around, taking up space that intended customers would otherwise use.