New API: Missing Current Condition Narrative and City of PWS

  • 3
  • Idea
  • Updated 5 months ago
Been working with the new api.  A couple of things clearly missing are as follows.  I

1.  Can not return the  Current Condition narrative of the PWS  such as 'sunny' or 'cloudy'.  One can use the Forecast call to get a forecast condition.  In my case using the latlong coordinates of a PWS.  However forecast 'narrative' could very much be different than the current condition surrounding a PWS

2. Can not return the City of the PWS.  

Both of these seem fundamental to me and hopefully, they can be added in the future
Photo of VG

VG

  • 1 Post
  • 0 Reply Likes

Posted 5 months ago

  • 3
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 637 Posts
  • 91 Reply Likes
I will pass this along.

Victoria Gardner
victoria.gardner@ibm.com
Photo of Joel

Joel

  • 6 Posts
  • 3 Reply Likes
I was playing more with the forecast and it doesn't look like it provides any type of current weather observations like what VG is asking (and what I was using previously). The closest is the forecast for the "daypart", which is a forecast for that 12 hour period (and provides an icon code and narrative text).

Would it be possible to get current, observed weather conditions as both text (so sunny, cloudy) and the icon id when using the https://api.weather.com/v2/pws/observations/current endpoint?
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 637 Posts
  • 91 Reply Likes
Keep in mind that a PWS, in the design of this API, does not have a forecast.  Forecasts are for larger areas.  So it was not designed to call forecasts by PWS.

Current conditions are data rather than narrative.  You can work out your own narratives, if you like.  Are you asking, Joel, for some kind of automated process to turn the current conditions from an individual PWS into a narrative?  

If you know what the icon would be -- sunny, cloudy, etc. -- why not just write the code to say it yourself if you need to display it?  I guess I'm missing something, I don't understand why this text is necessary when the information is available.  

--Victoria
Photo of Joel

Joel

  • 6 Posts
  • 3 Reply Likes
Sure, let me describe my use case. I have a little dashboard where I display my PWS data such as temperature and some other data from around my property. The old API would return PWS data, along with observed conditions such as sunny, rainy, etc which I would display on my dashboard.

While I could probably figure out if it is sunny or rainy from the data (solar radiation and precipitation rate), I wouldn't be able to determine the type of precipitation, or even accurate cloud cover. That type of data I would assume would be derived on your end from sources such as satellite and doppler, or you have some modeling based off of experience of looking at lat/long along with time of year to determine what solar radiation would look like for sunny/cloudy.

Other APIs I've looked at (such as Dark Sky) also provide some type of observed, current conditions. In the API offered to PWS owners, this is missing.
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 637 Posts
  • 91 Reply Likes
I'll pass it along!

--Victoria
Photo of John KUBIATOWICZ

John KUBIATOWICZ

  • 9 Posts
  • 1 Reply Like
I'd like to put in a word here as well.  The loss of current conditions in a location (integrating weatherstation data together with whatever other information is available in the current location) is very unfortunate.  In particular, I am also looking for an icon number and description of the current situation.  It appears from the Icon documentation that Icons can be returned from "observations" (whatever that means).  Can we get them somehow?  (I don't want to have to process my weatherstation data to derive a consistent Icon).
(Edited)
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 637 Posts
  • 91 Reply Likes
Okay, so perhaps what we have here is an incomplete understanding of what a "forecast" is.

If you spend time watching the forecasts, you will notice they are not fixed; they change over time.  I guess a lot of people out there in the world (not any of us weather geeks, but "them, out there") look at a forecast once or twice a day and imagine it as unchanging all the rest of the time.

But it does change over the course of the day.  Most of the time, the near forecast -- what's going to happen today -- changes relatively little.  But sometimes, there is so much variability in the models that the forecast can change hourly during the day, both for the current day and the other near-days, and of course further out in a longer term forecast.  I know I check the weather forecast multiple times per day.  It's not that I've forgotten it since I last looked, but that it might have changed.

I do not know how often the forecasts in the API change, but I would not make the assumption that they do not change over the course of the day.  The "current conditions" is by design limited to one point in space, where the PWS is located, so does not have the cloudy/sunny etc icon in it, because that information is for a much larger space.  To add the cloudy/sunny information to the current conditions API would be introducing the possibility of error, since there is no way for that particular PWS to report it.  UV data, yes.  Pressure, temp, wind speed, yes.  But is it partially cloudy, thunderstorming, etc?  Nope, that PWS doesn't know it, except as that information can be derived from its data.

Which means, to me, that the current day of the 5-day forecast is usually going to be what the meteorologists predict is happening now.  And I would imagine that means the icon presented there is going to closely mirror, to the extent knowable, the current conditions of a particular area as a whole.

This design is different from how old WU understood its data, apparently.  And I know it's hard to reshape your thinking when you're used to a different pattern.  But this design has a lot to recommend it, once you get used to the new way of thinking.

--Victoria
Photo of Joel

Joel

  • 6 Posts
  • 3 Reply Likes
No disagreements here Victoria, but it is still a shame to see the current conditions disappear. I'm assuming in the old API, when you queried by PWS, it would just pull in the conditions for a more wider area where that data was available/reported for. That was perfectly fine for my case, and obviously I understand there's no way for my PWS to report the current conditions (which is why I am looking for it). Forecasts don't really cut it because I don't want my dashboard to show the thunderstorm icon if there are thunderstorms forecasted for late in afternoon, but currently it is sunny. Even the daypart of the forecast isn't enough because it covers 12 hours.

For now, I'm just pulling current conditions for my area with a different API provider. I would love to see this functionality return though as part of the PWS provided API since it provides a nice display.
(Edited)
Photo of John KUBIATOWICZ

John KUBIATOWICZ

  • 9 Posts
  • 1 Reply Like
It is true that forecasts and current conditions are not quite the same thing, even if the forecasts are relatively short term.  As Joel points out, the most obvious difference comes up with respect to rain (is it actually raining, or is there a forecast for rain).  However, other things are more subtle, like is it cloudy or not (and how much). Clearly, the weather station data from a single weather station is insufficient to figure out this information (I would say that requires weather modeling + input from multiple weather stations).
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 637 Posts
  • 91 Reply Likes
But you do have UV and precip on current conditions.  So it should be possible to use those to replicate it.
Photo of John KUBIATOWICZ

John KUBIATOWICZ

  • 9 Posts
  • 1 Reply Like
It sounds like you are suggesting that we somehow duplicate the functionality of The Weather Company -- which has access to lots of weather data that we do not have.  I only have one weather station. My neighbor has one.  Our neighborhood and city have many.  So -- someone with the right access and models could get a good sense of current conditions in a region (like say a complete zip code or city).

Forgive me if this sounds snarky, but the point of having the API to do that for us is that we don't have to write our own weather modeling to glean current conditions over a small region from the data of multiple PWSs.  Also, given the limits on queries, it would seem that querying all the local PWSs to get a good sense of the current conditions might run afoul of limits. 
Photo of John KUBIATOWICZ

John KUBIATOWICZ

  • 9 Posts
  • 1 Reply Like
p.s. In California, here, we are the pinnacle of micro-climates.  I see less rain than my neighbor.  I am interested in the status of my city (or zipcode) rather than just my weatherstation.  Herein lies the challenge... 

Just out of curiousity, is it not possible to run an "extraordinary" short-term forecast that produces the current conditions?  What happens to your models when the expiration time approaches?  (or perhaps it doesn't work that way -- I'm not anything close to a weather expert).
Photo of John KUBIATOWICZ

John KUBIATOWICZ

  • 9 Posts
  • 1 Reply Like
p.p.s Aren't there lots of people who have weatherstations that don't produce rainfall totals?  Doesn't that mean that they no longer get information on current conditions about whether or not it is raining in their region or not?
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 637 Posts
  • 91 Reply Likes
Well, and as I said, I personally would probably just use the history or the forecast and call it a day.  I still have not understood the reason for needing that narrative version for a home user.  It seems to me that you're pretty close -- that everyone who wants it wants it to duplicate the forecast.

Recent history calls will come close to duplicating current conditions.  

It's unlikely that the PWS API is going to provide all the data that you would need to make your own version of a Wunderground page.  It's to provide data that people need for their own personal use.  

--Victoria
Photo of John KUBIATOWICZ

John KUBIATOWICZ

  • 9 Posts
  • 1 Reply Like
Ah.  Perhaps I haven't understood the "recent history" portion of the API.  I will look at it.

I can't really speak for everyone here, but I think that most of them just want something that says what the current conditions are in a queried region (like the one they requested the 5-day forecast on).  For many of them, this is almost as interesting as the forecast.  It says "what's going on in my town right now?"  It is the cool thing that kids (and adults alike) can see from a query (look -- I'm able to find out that there are clouds right now with this rasberry-PI!).  As you have seen from the blog traffic, the Wunderground served a lot of communities other than just the single home user -- schools, small-time home automation providers, hobbyiests.  I think that people are looking for the equivalent of the current conditions fields of the original WU API.  Something that doesn't require a lot of post-processing to turn into a graphic. 

It is also something that seems like it ought to be really easy to provide if you are already doing the (much more complex) provisions of 5-day forecast. 

My $0.02.  Perhaps
Photo of Joel

Joel

  • 6 Posts
  • 3 Reply Likes
Agreed. I just started to update my personal scripts and I ran into this same issue.

VG, it looks like you can use the location search utility API to look up your PWS ID and return a place id. Then you can use that place id when getting the forecast, which hopefully would be more accurate to your PWS. No idea if this new API (or the old one really) updated the actual observed weather conditions to be that precise :-)
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 618 Posts
  • 88 Reply Likes
Yes, Joel.  Forecasts are pretty much never as precise as a single PWS.  Thank you!

--Victoria
Photo of Claude Felizardo

Claude Felizardo

  • 30 Posts
  • 6 Reply Likes
How often do we need to look up the "place id"?  Is this "place id" going to be the same for a given PWS or could it change every once in a while?  Daily or weekly or hourly?

My concern is having to make several API calls that will eat into our minute or daily limits. 

If it does change periodically can we look it up at startup of our program, use a cached value until we get some kind of error indicating it is no longer valid then do a new lookup only as needed?

Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 618 Posts
  • 88 Reply Likes
I hear you about the calls, Claude!

If you're calling your own PWS, I guess I would be inclined to use the PWS ID where possible, because you will have to know if that changes.  (Because you would have to be the one changing it.)  Similarly if you're calling someone else's PWS, the ID should fail if it's unknown, but with something like a lat/long, it may do the best it can and give you something you don't want.  

Generally a PWS isn't going to move except under unusual circumstances.  But they do come and go.  

If you have a particular place you like, you could set up something like this.  Call the PWS by ID (if you can), and then if that fails, do the "near" call and see if you can find something similar close by.  I would only do the searching/revalidation "on error".  So if everything is fine, go ahead, else, revalidate/refind.  

Does that make sense?

--Victoria
victoria.gardner@ibm.com
Photo of Claude Felizardo

Claude Felizardo

  • 30 Posts
  • 6 Reply Likes
yes.  makes sense.  I would only be interested in my own PWS, again to make sure it is uploading correctly. 

If I get a chance to fix a package that could be used by others then I would keep do the steps you outlined above. 
Photo of grahamck

grahamck

  • 5 Posts
  • 0 Reply Likes
If I look at the WU page for my PWS I can see a "current conditions" attribute and this is the only thing that I am missing from the previous API that I really would like to have. I can get it from openweathermap and inject it into my home automation system, but I sure would love to be able to get it from WU. Am i missing this someplace in the new API?  

In case you are curious of the possible use cases here is one of many - my home automation system will turn off image based motion detection on my UV cameras when it is snowing at night. Previously I would get the condition "Snowing" from the WU API - not perfect, but pretty darned good.....

Otherwise it appears that I can get everything I need from the new API. Forecast precipitation being the biggie with irrigation season coming up.......

Thanks.

(Edited)
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 637 Posts
  • 90 Reply Likes
Graham, I'm really not sure what you're looking at.  When I look at a PWS dashboard, I don't see anything other than what's in the PWS API's "current conditions" API.  The forecast for the PWS has other things, but that's not PWS specific.

I guess if that were my only use case, it'd be:
if temp <= 35F $and$ precip>0 for the current hour, shut off camera.  Not perfect, but neither was the old current conditions.  

Someone else in one of these questions asked about how you tell snow from rain, and I'm not sure even I ever worked out an answer to that.  But there must be one.

--Victoria
Photo of grahamck

grahamck

  • 5 Posts
  • 0 Reply Likes
Measuring rainfall, unfortunately does not work for snow. I'm looking at the circled (in red) portion of the page that comes up for my PWS:


The "current conditions" are "cloudy". Am I missing that somewhere in the API? I cannot find it. I rely on this for a fair amount of things.....  Thanks!
(Edited)
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 637 Posts
  • 90 Reply Likes
Ah, yes.  That is aggregated data, which is not on the free PWS API.  And I'm pretty sure it's not going to be in the foreseeable future. 

--Victoria
Photo of grahamck

grahamck

  • 5 Posts
  • 0 Reply Likes
That's too bad and will make me move to another service to get this data. Not sure what my motivation is to provide data to WU these days.
Photo of John KUBIATOWICZ

John KUBIATOWICZ

  • 9 Posts
  • 1 Reply Like
Victoria, there seems like a lot of loss for folks with the new free API.  Isn't the point to keep providers of the data happy?  Why the major shift?  The whole point of WU was crowd-sourcing: Add your bit of data and get interesting aggregated information like forecasts and current conditions. 

I looked at the "historical" API as you mentioned, and it doesn't see to provide anything like current conditions (in the form of the icon/text indicating what is going on).  It is just a historical list of data from a weather station.  Is there some business model behind not providing limited aggregation to users? 
Photo of grahamck

grahamck

  • 5 Posts
  • 0 Reply Likes
^^^ exactly ^^^
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 637 Posts
  • 90 Reply Likes
Keep in mind you're talking about WU as it was before.  And then it was sold, and that company was sold, and now it's in a whole different context.

I am not privy to conversations about business models.  However, I do get a clear sense that the aggregated data is considered highly valuable.  And if users want it, they need to pay for it.

"Add your bit of data and get interesting aggregated information like forecasts and current conditions."  But this is exactly what we're doing!  

I've said this elsewhere, but switching APIs is like learning a new language.  Things you were used to saying one way before cannot be adequately translated.  As far as I can tell, there is an obsession here with this faux-current conditions narrative from before that flies in the face of reality.  It was never "current" at one spot.  For a while the iPhone app had a slider so you could send in what was reality at your location (whether or not you had a PWS).  It was almost never perfect where I was.  

If you want to know what the current conditions are, go to WU, on a browser or in an app.  I don't understand why a very small subset of users are so insistent that we provide narrative "conditions" through the API for free, when we're providing it on the branded products that everyone already has access to.  I have offered any number of suggestions for working around this in general.  The data to create your own version of Weather Underground using our own data just isn't going to happen anymore.  Maybe that's why the prior incarnations of this business were not successful.  I don't know.  But it sure does seem, from my perspective, that they opened the business model up for serious abuse, and IBM seems to want to learn from that lesson.

I'm sorry you're unhappy about this, I truly am.  But this is the reality that we're in.  

--Victoria
Photo of grahamck

grahamck

  • 5 Posts
  • 0 Reply Likes
In my case you totally and completely misunderstand your user base. A computer application cannot "go to an app on it's iphone and look up the conditions". If you consider "current conditions" as the data that I am sending you, then rest assured that sending you that data is highly optional for me. There are plenty of ways for me to understand the data I am sending you, and a myriad of places that I can download weather forecasts for use by algorithms that serve a lot of functions. When it comes to "is it raining, or is it snowing", these are never absolutely perfect, of course not. But they are close enough for the work that I use them for. I am currently getting that data through OpenWeatherMap since you no longer provide it. No longer providing that key piece of information means that I will no longer be using WU, which means that i have very little reason to send you my data anymore. I'm not sure if the new corporate model really cares if I send you data that you intend to aggregate with other data and attempt to sell back to me? But that will be the eventual outcome.  Hopefully that is clear, not partly cloudy, pun intended. ;-)

Really the only thing left for me to do is tell my weather station to stop sending you data. <shrug> TBH - it's annoying, but really not that big of a deal to have to change how I collect data to a method that doesn't include WU.
(Edited)
Photo of Bees

Bees

  • 4 Posts
  • 1 Reply Like
It's sad that we can't even get the current condition and corresponding icon. Simple to allow us. Not asking too much. I use it for my personal weather wallpaper and that section is missing (as shown). C'mon WU, bend a little for us!

Photo of John Kubiatowicz

John Kubiatowicz

  • 1 Post
  • 3 Reply Likes
Victoria, IBM seems to have a fundamental misunderstanding of their data providers.  Monetizing a crowd-sourced platform is tricky.  People have many reasons for choosing to contribute their data.  Perhaps IBM should do a bit more to understand their data sources?

People you are hearing from are paying the weather company with data to get a service.  They purchased weather stations, wrote custom upload code in some cases, and are paying for internet service, power, and maintenance costs to maintain a feed of data to IBM.  They are paying with the currency of data.  What do they want for this payment?  That is the question that IBM has not really understood, in my opinion. Instead, you are responding in a condescending tone and suggesting that all these people are unable to deal with the new language of the API and were all previously abusing the platform.  You are confusing syntax (language) with semantics (processed results).

Why do people contribute data?  Well, for the aggregated results of thousands of data providers (semantic results in the form of processed data), not for the unaggreggated data from their own weather stations. Any programmer worth their salt can incorporate a database into their weather feed.  Thus, the historical queries are not really all that amazing, nor is the current status of their data feeds.

You have presented a fundamental paradox, which reveals your actual position.  Either the Current Conditions Icon is worth very little (so much that clearly people can reproduce it by downloading their own data) or it is worth so much that IBM is unwilling to give it out to peons ("you are not going to be able to duplicate the WU home page").  Which is it?  Despite what you may or may not personally believe, the reason many people go to the trouble of uploading their data (and the cost!), is to get the aggregated current conditions.  Others want a forecast.  Both of these things were part of the value proposition that caused people to chose to upload their data feeds. 

I fully believe that the query traffic overloaded the servers that IBM chose to devote to WU.  I fully believe that the free API was used by lots of people that were not paying at all (not even in the form of data).  However, treating your data-paying customers like they are trying to cheat the system and get something for free or something that they shouldn't reasonably expect without paying IBM a bunch of money is both insulting and reflective of not understanding what motivated people in the first place. People may just take their data elsewhere.

Now you (mostly) seem like a reasonable person, so I have to assume that you have neither the power nor inclination to try to investigate what your data-paying customers want out of the crowd-sourced system.  Such is the state of those hired to deliver an open-loop message without any chance to incorporate feedback.

Personally, this seems very sad.  I always liked to hold up the Weather Underground as one of my favorite examples of a crowd-sourced platform.  I like to talk about it to our industrial partners -- even foreign ones.  This will no longer be the case.

As a closing thought, I suggest that IBM consider what the cost might be (to them) of providing the current conditions ICON/text as part of the "free" (to data-paying customers)  5-day historical query and decide whether the data feed might be sufficient to pay for a current condition icon. And, I would ask that you treat those on this forum as PAYING customers -- people paying with data.

That might have been a whole dollar worth of commentary, to which I'm sure you don't care.

So long and thanks for all the fish.
(Edited)
Photo of Carson Booth

Carson Booth

  • 1 Post
  • 0 Reply Likes
This is the most intelligent description and response of the situation I have seen to date. Thank you and as a data payer for years (IBerlin1622), I have spent the last 2 weeks porting over to DarKSky for this exact reason described above.
Thank you and sad to exit WU in the coming weeks.
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 637 Posts
  • 91 Reply Likes
I have put the request for this change in the queue to be analyzed and decided upon (by people other than me) in due course.  
Photo of elee

elee

  • 11 Posts
  • 1 Reply Like
Victoria, please help us to obtain this simple information. Maybe you think it was something vague, but in my case it was the most accurate data of any other weather system on the market. Please, we only need that information to be able to continue with WU. It is a request that abounds throughout this forum. Do not make us go to other services. Let us all make WU even better. Please.
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 637 Posts
  • 91 Reply Likes
I will pass the request on to the people who make such decisions.

--Victoria