SSL Certificate Error

  • 5
  • Question
  • Updated 4 months ago
Unable to upload data to WU due to ssl.CertificateError.  Started at 2300hrsGMT last night (29 Jan 20).

No changes to my system at all, just wondering of any of the changes currently happening within WU's systems have caused the issue?
Photo of David McCreath

David McCreath

  • 6 Posts
  • 0 Reply Likes

Posted 4 months ago

  • 5
Photo of Brian Mallard

Brian Mallard

  • 3 Posts
  • 0 Reply Likes
Just an FYI, but I starting getting a certificate error from my weewx/raspberry pi a couple of days ago. also looking for information.

Jan 30 11:35:15 raspberrypi weewx[583]: restx: Wunderground-PWS: Unexpected exception of type <class 'ssl.CertificateError'>
Jan 30 11:35:15 raspberrypi weewx[583]: restx: Wunderground-PWS: Thread exiting. Reason: hostname 'weatherstation.wunderground.com' doesn't match either of '*.prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'
Photo of Mark Kimble

Mark Kimble

  • 3 Posts
  • 0 Reply Likes
OH LOOK, IM not alone.
Photo of Tim Roche

Tim Roche, Official Rep

  • 356 Posts
  • 53 Reply Likes
working on this.  Expect resolution within a few hours.  
Photo of Phil

Phil

  • 3 Posts
  • 0 Reply Likes
I'm also having the same issue.  I have been working on my system thinking it was just me.  Are you guys still down?
Photo of Tim Roche

Tim Roche, Official Rep

  • 356 Posts
  • 53 Reply Likes
Please add any errors you see, we think we have this resolved, but it seems a handful of stations remain with issues
Photo of Brian Mallard

Brian Mallard

  • 3 Posts
  • 0 Reply Likes
as of midnight EST 1/31 I am still seeing errors when trying to upload from weewx on a Rasbberry pi 

Jan 31 00:00:30 raspberrypi weewx[583]: restx: Wunderground-PWS: Failed to publish record 2020-01-31 00:00:00 EST (1580446800): Failed upload after 3 tries

Jan 31 00:11:52 raspberrypi weewx[4321]: restx: Wunderground-PWS: Failed upload attempt 1: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)>
Jan 31 00:11:58 raspberrypi weewx[4321]: restx: Wunderground-PWS: Failed upload attempt 2: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)>
Jan 31 00:12:03 raspberrypi weewx[4321]: restx: Wunderground-PWS: Failed upload attempt 3: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)>
Jan 31 00:12:08 raspberrypi weewx[4321]: restx: Wunderground-PWS: Failed to publish record 2020-01-31 00:10:00 EST (1580447400): Failed upload after 3 tries

(Edited)
Photo of Tim Roche

Tim Roche, Official Rep

  • 356 Posts
  • 53 Reply Likes
Yuck, very frustrating.  Can you determine what URL endpoint you are trying to send to?  https://rtupdate.wunderground.com/ and https://weatherstation.wunderground.com/ both show valid certificates  This use this in your browser to test the certificate https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php?ID=<your id>&PASSWORD=<your upload key>&dateutc=now  You should not see cert errors, and hopefully you will see "success" I have validated that from multiple locations this works, but obviously not for some people.  We're working hard to get this resolved for you.  Couple things to think about.
Make sure you are not using a cached DNS entry, the wu dns ttl is 60 seconds, ensure you obey this.  I see a very significant number of stations still sending to our old endpoints telling me they are somewhere downstream from a bad DNS server (possibly theirs, possibly not)
restart your computer/router/modem.... whatever this is connected to, that could be caching something bad.    
try to get a full trace of your connection, including what the certificate error is
Photo of Cameron Davidson

Cameron Davidson

  • 1 Post
  • 0 Reply Likes
Both short urls return "404: not found" using firefox - https or http.
Edit: OK, forget this, because the longer url with auth data inserted  does return certificate results.

Both resolve as cnames to
prod-pws-ng-ingest.prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud
which is a CNAME to
prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud
which resolves to 3 A records:
169.47.111.58,
52.116.188.166,
and 169.60.133.170
All of which have tiny TTLs

This is after pushing the lookup through the root servers, so I don't think there are any cache issues.

I also am using weewx, like some of the other reporters.  I am also currently testing a beta version, so I need to look more closely at where my logs are disappearing..

Also, I noticed one other reporter looks to be in Australia, like me - not sure if that is relevant

(Edited)
Photo of David McCreath

David McCreath

  • 6 Posts
  • 0 Reply Likes
Tim

SSL Certificate error now solved my end.  However, like Brian, WU is still unable to receive pushed data sent from my system:

Jan 31 07:35:41 raspberrypi weewx[543]: restx: Wunderground-PWS: Failed to publish record 2020-01-31 07:35:00 GMT (1580456100): Failed upload after 3 tries
Jan 31 07:40:32 raspberrypi weewx[543]: restx: Wunderground-PWS: Failed to publish record 2020-01-31 07:40:00 GMT (1580456400): Failed upload after 3 tries
Jan 31 07:46:26 raspberrypi weewx[545]: restx: Wunderground-PWS: Failed to publish record 2020-01-31 07:45:00 GMT (1580456700): Failed upload after 3 tries

Camera upload working fine though.

RPI rebooted several times and I have the same error on either Rapid Fire or timed data uploads.
(Edited)
Photo of Mark Kimble

Mark Kimble

  • 3 Posts
  • 0 Reply Likes
Jan 31 03:05:49 <snip> weewx[5197]: restx: Wunderground-RF: Failed to publish record 2020-01-31 03:05:43 EST (1580457943): Failed upload after 1 tries

repeated regularly.  I flushed my dns cache.  but alack, alas, and Alaska.

Will ahve to play in the morning and see if it is still me.  Though Im thinking not.

thanks for your work!

Photo of Peter Clarke

Peter Clarke

  • 2 Posts
  • 0 Reply Likes
Same error for me.
Started at 2020-01-29 23:00 UTC


raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='weatherstation.wunderground.com', port=443): Max retries exceeded with url: /weatherstation/updateweatherstation.php?ID=IBRISB670&PASSWORD=#&dateutc=now&humidity=77.00&tempf=77.90&action=updateraw (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

appreciate the help
Photo of David McCreath

David McCreath

  • 6 Posts
  • 0 Reply Likes
With WU having fixed the ssl certificate error last night (was still unable to upload data though), as of an hour ago, the error is back:


Jan 31 10:50:27 raspberrypi weewx[545]: restx: Wunderground-PWS: Thread exiting. Reason: hostname 'weatherstation.wunderground.com' doesn't match either of '*.prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'



Bummer.
Photo of Phil

Phil

  • 3 Posts
  • 0 Reply Likes
I tested my certificate as instructed and do receive a "success" code within a browser.  However, I have tried both addresses from my weather station and still get the following error.  I have also tried rebooting my station (multiple times), and rebooting the router.  This is ran on a raspberry pi, so I do not believe there is a DNS cache to clear/flush on the pi/station its self.
(Edited)
Photo of Dave Albright

Dave Albright

  • 1 Post
  • 0 Reply Likes
also tested my certificate as instructed and do receive a "success" code within a browser.
currently receiving:
restx: Wunderground-PWS: Failed to publish record 2020-01-31 06:30:00 MST (1580477400): Failed upload after 3 tries
and eventually
restx: Wunderground-PWS: Thread exiting. Reason: hostname 'weatherstation.wunderground.com' doesn't match either of '*.prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'

Photo of Joshua Myles

Joshua Myles

  • 4 Posts
  • 0 Reply Likes
Just looking at weatherstation.wunderground.com, it's not passing back the full certificate chain, just the *.wunderground.com wildcard. Intermediate and CA certs should also be passed in order for non-browsers (like curl or python) to work.
Photo of Joshua Myles

Joshua Myles

  • 4 Posts
  • 0 Reply Likes
This should pass when the certs are configured correctly:

$ openssl s_client -showcerts -servername weatherstation.wunderground.com -connect weatherstation.wunderground.com:443
CONNECTED(00000003)
depth=0 C = US, ST = Georgia, L = Atlanta, O = "The Weather Company, LLC", CN = *.wunderground.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = Georgia, L = Atlanta, O = "The Weather Company, LLC", CN = *.wunderground.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:C = US, ST = Georgia, L = Atlanta, O = "The Weather Company, LLC", CN = *.wunderground.com
   i:C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
-----BEGIN CERTIFICATE-----
MIIGtjCCBZ6gAwIBAgIQD65sC/9G3gQoGmCGLLT/qTANBgkqhkiG9w0BAQsFADBN
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMScwJQYDVQQDEx5E
aWdpQ2VydCBTSEEyIFNlY3VyZSBTZXJ2ZXIgQ0EwHhcNMTkwODIxMDAwMDAwWhcN
MjEwODI1MTIwMDAwWjBxMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHR2VvcmdpYTEQ
MA4GA1UEBxMHQXRsYW50YTEhMB8GA1UEChMYVGhlIFdlYXRoZXIgQ29tcGFueSwg
TExDMRswGQYDVQQDDBIqLnd1bmRlcmdyb3VuZC5jb20wggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQCZaE//9x/jjc91iL5mr7d+PMFXOrQ8Z7QQL/ceaNxr
/evw9Yr2XRcMzCCIaz04O+Sr19ti3wwBPIBzngQxCWq5P9Qsi4F1DXmyXTbDoma5
uSMdN+QOaSVPk32XZrA7tS68lEURZwtWfBu4YcI6/jacuJ2V6H1JqOaFRzgzssOx
+LPOz4vHQGDAvK33c+svh5ULiOpJB0XVYKGZ4G0SrMVlipMjyH+rA662sZwzP9pN
83OrPoOTj0pL/Ui9DqR74eAeY5rk3O85FnjjUzuNR72NcqnqALhsBMvmmYxAEhEw
pUcPqu0HbZFzYWHjMYix8oB/dYgieDk1BbnBvi0yxUBfAgMBAAGjggNsMIIDaDAf
BgNVHSMEGDAWgBQPgGEcgjFh1S8o541GOLQs4cbZ4jAdBgNVHQ4EFgQUVv18U4QK
+c8znzL4KztKffVoUUYwLwYDVR0RBCgwJoISKi53dW5kZXJncm91bmQuY29tghB3
dW5kZXJncm91bmQuY29tMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEF
BQcDAQYIKwYBBQUHAwIwawYDVR0fBGQwYjAvoC2gK4YpaHR0cDovL2NybDMuZGln
aWNlcnQuY29tL3NzY2Etc2hhMi1nNi5jcmwwL6AtoCuGKWh0dHA6Ly9jcmw0LmRp
Z2ljZXJ0LmNvbS9zc2NhLXNoYTItZzYuY3JsMEwGA1UdIARFMEMwNwYJYIZIAYb9
bAEBMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMw
CAYGZ4EMAQICMHwGCCsGAQUFBwEBBHAwbjAkBggrBgEFBQcwAYYYaHR0cDovL29j
c3AuZGlnaWNlcnQuY29tMEYGCCsGAQUFBzAChjpodHRwOi8vY2FjZXJ0cy5kaWdp
Y2VydC5jb20vRGlnaUNlcnRTSEEyU2VjdXJlU2VydmVyQ0EuY3J0MAwGA1UdEwEB
/wQCMAAwggF9BgorBgEEAdZ5AgQCBIIBbQSCAWkBZwB1ALvZ37wfinG1k5Qjl6qS
e0c4V5UKq1LoGpCWZDaOHtGFAAABbLV7gUQAAAQDAEYwRAIgfVt598CsSlGixHNn
3m5jtU/fmuO0Snvy1OtiksZd/XwCIBeDk77NI/NYW32FB6IWMKoNJIpK/3eHclL5
RGuqdqPjAHYAh3W/51l8+IxDmV+9827/Vo1HVjb/SrVgwbTq/16ggw8AAAFstXuB
jAAABAMARzBFAiAPmstixdjNaG9sboiKmoA+Gj1vuj7YIu7FMXPRYgYNiQIhANZZ
NJHoA7r5rmQQ3q3gFP5IHGwy1gsu7AyZPyVgfMYaAHYARJRlLrDuzq/EQAfYqP4o
wNrmgr7YyzG1P9MzlrW2gagAAAFstXuAlQAABAMARzBFAiA2qOttc3uXexk9dZJ8
MXptc+xpChi7K0rJBNLqJ652KwIhAKcUDVSYI7CxYzdmIJeXdNYBEkNyDOBZDBrx
uR0CSIkjMA0GCSqGSIb3DQEBCwUAA4IBAQBGrOqSFbw7OseR+MpMWS2gb9a4lzYA
XhmiyxaDrrz8UfDf0/BG8VX4Zst0DEl4xlTxCUGnj2v34t7e1a+yUDmAZeBVd7of
Qe8jKInBZiY7r+YnrTo+kMoMLpSv2OHNEGO2N7ZRMWjD0ECmVkuSw+TYgyx/n3kJ
lcsahB6pXi8KNpi0jWo1DZYCirc31GfzlX3NIktsCJqPKH3Pq9JsSx4/4dI2TUJ3
ekcXVBjczDqXqtULv5kTE8BcHZtGdFFk1oDFbo0xDzPobs+7ocOAfiw7kZWFGnSJ
y56BbPdb6VUM6ant8mcCuCXORD668r8W9W//TZLJZv+0LbOg5/6a6VtG
-----END CERTIFICATE-----
---
Server certificate
subject=C = US, ST = Georgia, L = Atlanta, O = "The Weather Company, LLC", CN = *.wunderground.com

issuer=C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2282 bytes and written 413 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
...
$


(Edited)
Photo of Brice Ruth

Brice Ruth

  • 5 Posts
  • 0 Reply Likes
You can see this from digicert's SSL checker, too - https://www.digicert.com/help/
Photo of Tim Roche

Tim Roche, Official Rep

  • 356 Posts
  • 53 Reply Likes
We think this is resolved
Photo of Clint Satterwhite-W5CMS

Clint Satterwhite-W5CMS

  • 3 Posts
  • 0 Reply Likes
I just tested by reverting my workaround and it seems that my packets are now accepted via HTTPS again. Thanks!
Photo of Joshua Myles

Joshua Myles

  • 4 Posts
  • 0 Reply Likes
I agree, this is fixed for me too. Verified that I'm getting full cert chains for weatherstation and rtupdate.
Photo of Dana Borgman

Dana Borgman

  • 4 Posts
  • 1 Reply Like
Logging observations here as of 2:19 EST.  Thank you.
Photo of Clint Satterwhite-W5CMS

Clint Satterwhite-W5CMS

  • 3 Posts
  • 0 Reply Likes
Since there are no solutions posted yet, as a workaround I edited /usr/share/weewx/weewx/restx.py and replaced the instances of https with http in the rf_url and pws_url variables. This got my station back online.
Photo of Peter Clarke

Peter Clarke

  • 2 Posts
  • 0 Reply Likes
Yep. Back online and uploading. 
Thank you.
Photo of David McCreath

David McCreath

  • 6 Posts
  • 0 Reply Likes
Tim

Many thanks, all back up and working my end.
Photo of Brian Mallard

Brian Mallard

  • 3 Posts
  • 0 Reply Likes
Looks like I'm back online as well. Thanks for your help
Photo of Paul Hutch

Paul Hutch

  • 21 Posts
  • 7 Reply Likes
All my stations are working again too. Big thanks to Tim and the rest of the WU staff.
Photo of Clint Satterwhite-W5CMS

Clint Satterwhite-W5CMS

  • 3 Posts
  • 0 Reply Likes
@tim_roche 
And... its broken again. I re-instituted my workaround.
(Edited)
Photo of David McCreath

David McCreath

  • 6 Posts
  • 0 Reply Likes
I've had two drop outs so far this morning, both SSL Certificate related.
Photo of Crispin Bennett

Crispin Bennett

  • 1 Post
  • 0 Reply Likes
I am back online (Raspberry Pi running PYWWS). Had done nothing other than multiple reboots. It appears to have uploaded all the missing data too (other than the gaps during rebooting). Thanks all.
Photo of Mark Kimble

Mark Kimble

  • 3 Posts
  • 0 Reply Likes
hmm.  still died after a few passes on me.  Maybe chaching.  I went all unsecure and started using http instead of https and works fine.  Im going to sit around and hope ttl on what ever is causing me grief expires.
Photo of tmarschner

tmarschner

  • 3 Posts
  • 2 Reply Likes
Same for me. Dies after a few passes after a reboot on my Raspberry pi.
Photo of Jason Haley

Jason Haley

  • 1 Post
  • 0 Reply Likes
Also receiving ssl cert error after a few minutes of successful data uploads running weewx on raspberry pi.
Photo of Brice Ruth

Brice Ruth

  • 5 Posts
  • 0 Reply Likes
Same here. I think there’s still servers, gateway endpoints, or load balancer groups in the DNS rotation that aren’t setup right. Eventually you hit them. I hope the staff are aware and continue working on fully fixing the issue.