Dayparts go to Null at 3:00pm... but when are they refreshed for the new day?

  • 1
  • Question
  • Updated 8 months ago
I am at the last stage of converting my system over to the new API. I have worked around the issue of the Dayparts going to Null after 3:00pm... not that I like the cutoff time... I would prefer it to be later in the day! 

However, last evening, just after midnight, as I was testing my system I started encountering issues because the Dayparts did not start reporting for the new day! I built in the logic to deal with after 3:00pm of the current day, assuming that immediately after midnight the Dayparts would be refreshed for the new day!

My questions are:

- when does the system refresh for the new day; and

- is the refresh done based on local time?

Thanks
Photo of Gary

Gary

  • 5 Posts
  • 0 Reply Likes

Posted 9 months ago

  • 1
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 658 Posts
  • 93 Reply Likes
Yes, the refresh is based on local time.  It typically happens between 3-4 am.

Victoria Gardner
victoria.gardner@ibm.com
Photo of Gary

Gary

  • 5 Posts
  • 0 Reply Likes
Really?  "...typically happens between 3-4 am" ! Does it depend on when the person responsible for  flipping the refresh switch remembers to do his job?

I am surprised that a system like this doesn't have specific times for events to happen (like the 3pm to null)!

Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 658 Posts
  • 93 Reply Likes
No, it definitely is not someone sitting in front of a computer.  It depends on cache and other automated processes.  This is a part of a huge world-wide API.  

--Victoria
Photo of Gary

Gary

  • 5 Posts
  • 0 Reply Likes
Okay... I was kidding about someone sitting in front of a computer!!

I do recognize that I am only a single PWS user... however, I remain surprised that larger organizations, that rely on your  'huge world-wide API' data,  do not have a specific time specified for when the data is populated. Therefore they presumably have to program logic into their systems to continually look for that point in time when the data goes from null to a value! 
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 658 Posts
  • 93 Reply Likes
It doesn't have to be continually -- it's going to be about 3 pm/am, and they only need to pay attention to it until it changes state.  Or when they care about it.  Does that make sense?  It does to me, but perhaps because I've thought about it for some time already.  

--Victoria
Photo of Gary

Gary

  • 5 Posts
  • 0 Reply Likes
I am sure you have thought about it a lot more than I have... and I also don't want to keep beating the point... I still think it would be continually because I doubt someone would want to write code that  only tests for the refresh happening "about 3 pm/am"... Given the uncertainty of time, I expect that the test would be continuous.

Given this, I will rewrite my code to do likewise... Thanks
Photo of Claude Felizardo

Claude Felizardo

  • 32 Posts
  • 6 Reply Likes
So when this null error occurs, how often do we have to check?  We don't want to exceed any limits.  Check once a minute until we get non null or what?
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 658 Posts
  • 93 Reply Likes
Okay, it's not an error.  It's by design.

When you get a null for the forecast, don't keep checking it -- it means that the "day" forecast is over, as the "day" is pretty much over.  The API includes time in most calls, so you can verify what time it thinks it is.  Or the "night" part is over, and it's heading toward day.

I'm not sure why you'd want to check the forecast so often.  There are industrial uses that need to check often, but what is the use case for it from PWS users?  It's difficult to make suggestions when I don't understand what functionality you're reaching for.

Victoria Gardner
victoria.gardner@ibm.com
Photo of Claude Felizardo

Claude Felizardo

  • 32 Posts
  • 6 Reply Likes
Ok, got it.  From the original question and initial replies it was not clear when a null was being returned.  I just saw a reply that said to check again.  Your last reply makes it clear that a null will be used when the forecast for that part of the day is no longer valid and it will eventually contain data later.

I've got devices that periodically connect to WU to obtain current observations from my PWS and/or the forecast and display some information either as text or a graphic or something.  As I got about recoding things I'll keep an eye our for null.
Photo of Joel

Joel

  • 6 Posts
  • 3 Reply Likes
I also noticed that temperatureMax will begin to report null for index 0 (not under daypart). Any reason why this happens? I would expect to always be able to get the temperatureMax for the current day, regardless of time. Other parameters here at the root don't return null (sunrise, moonPhase, qpf, dayOfWeek, etc).
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 658 Posts
  • 93 Reply Likes
You're looking at the daily forecast, correct?

It's a forecast, not conditions.  The API is designed to eliminate the data that may no longer be relevant.  That data would then be accurate as historical information, not a forecast.

--Victoria
Photo of Joel

Joel

  • 6 Posts
  • 3 Reply Likes
It just seems arbitrary to return null at all. If the day is still going to be included in the dayOfWeek array, I would assume to find data for each of the properties for that day (and most due, just some just start displaying null). The old WU API didn't work this way, nor do other's that I've been looking at. This just seems like a really weird quirk and I'd categorize it as a bug personally.
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 658 Posts
  • 93 Reply Likes
You are certainly entitled to your opinion.  A bug, IMHO, is something that is not happening by design.  This API was designed this way.  Ergo, not a bug.  

Happily, we live in a world where difference is possible.

--Victoria
victoria.gardner@ibm.com
Photo of Paul O'Brien

Paul O'Brien

  • 2 Posts
  • 2 Reply Likes
I personally enjoy seeing UNDEF in big blue letters under Today's High on my wall mounted home automation tablets after some point in the afternoon. It goes outside the box of expecting to see regular old, boring numbers to indicate a temperature, and really impresses my guests when they walk in and can clearly see that the high for the day is, "UNDEF."
Photo of John KUBIATOWICZ

John KUBIATOWICZ

  • 9 Posts
  • 1 Reply Like
Toning down the rhetoric a tiny bit!

I'm sure that people are still trying to figure out the hows and whys of the new APIs.

I would second the fact that the presence of "null" entries does cause extra programming.  Since the outputs of the forecast are displayed in many panels around my house, I have to somehow remember the values that were formerly in the null entires so that they can continue to display properly when they don't exist in the polled data.  And, these cached values have to be invalidated if polling for the data fails during the early morning.

Not that I'm complaining too much, but depending on which language you are parsing the JSON in, the nulls in arrays of integers causes a bit of a headache.  (In C#, for instance, one needs to use arrays of nullable integers instead of arrays of integers, with the attached complexities).

I would also point out that the lack of precise cutover at midnight is going to cause some extra work as well, since it looks a bit weird when one checks after midnight and the "today" forecast is really from yesterday. For instance, I'm still not entirely sure what is true during the twilight hours after midnight but before the arrays refresh.  Which is true?

1) During the early morning, the "evening" forecast from the previous day is the valid one until the nulls go away.

2) At midnight, the "day" forecast for the new day is the valid one. 

Along these lines, the forecast does seem to evolve over the day.  How frequently should one poll to get the most interesting changes over the course of the day?
Photo of Ul KE

Ul KE

  • 13 Posts
  • 18 Reply Likes
There were another, still unanswered questions regarding the missing decimal numbers. The new new dashboard is only showing whole numbers which is not that, what we expect from a weather monitoring site. Most of us bought an expensive Davis Station and we need to see the detailled Temperature. Also the long term history is missing. It cannot be manually selected any more. These two point are really important to most of the private PWS Owners; Please comment. Thanks!
(Edited)
Photo of Victoria Gardner

Victoria Gardner, Official Rep

  • 658 Posts
  • 93 Reply Likes
Are you talking about the PWS dashboard online, or the API?

For the API, we have plans in progress to add precision to the temperature, which we hope to have rolled out by the end of April.  I don't know what the plans are for the web site/apps. 

Please explain what you mean by "long term history".  What exactly did you want to call and how would you use the data?  

Victoria Gardner
victoria.gardner@ibm.com
Photo of Ul KE

Ul KE

  • 13 Posts
  • 18 Reply Likes
Thanks for your answer.
Now I am talking about the PWS dashboard online. The decimal place is missing and the historical data is only accessable by selection day/week or month. In former times, you were able to select free timeframes. 
Therefore, the actual PWS dashboard is really not an improvement compared to the former one.