Wio Terminal App (rpcWiFi lib) sending telemetry data to Azure Storage Tables stops working after many uploads

RoSchmi · December 31, 2020, 11:50am

I’m working on an App which is sending telemetry data via secure https Post requests (rpcWiFi HTTPClient) to Azure Storage Tables every minute and actualizes system time every 5 minutes via NTP.
-https://github.com/RoSchmi/AzureDataSender_Wio_Terminal
Actually with RTL8720 firmware v2.1.0 and the actual libraries as of 31.12.2020 everthing works fine but after many successful uploads (more than 100, one time more than 500) the transmission stops as all successive post requests return with error code -1.
The program loop is still running and new tries to upload are initiated in the expected intervals.
Can anybody give some advice what might be the probable cause of this issue and how to debug and solve the issue?
(Memory leak? Race condition?)
Did anybody experience similar problems?

RoSchmi · January 4, 2021, 12:04am

I made some more investigations:

I synchronized SystemTime with the DateTime value which is returned in the Date header of the received response on my post requests.
-> Since the issue persisted the NTP requests used before cannot be the (only) cause.
I included a NVIC_SystemReset(); after a failed request:
-> The system rebooted as expected after a failed request and then performed normal operation for many uploads (up to about 300)
I included a watchdog in my application (I saw up to now no wathdog resets)
I activated the logging functions in rpcUnified (I observed no irregularities when I used https protocol)
I tried with http instead of https
Surprisingly I saw many failed requests (often after only a few successful operations).
-> In the logging output I could see that often the DNS request were not successful (error code -5)

Any ideas how to find the cause of the sporadic failed https post requests and why the App (without rebooting) doesn’t find back to normal operation after the first failed request?

Best regards
RoSchmi

RoSchmi · January 4, 2021, 10:40am

This time, when I had it running over night, I had 450 successful uploads, then there was a WatchDog Reset and the App continued to work after reboot.
The logging output seems to show that the start_ssl_client function didn’t return.
ssl_failure

lakshan · January 6, 2021, 3:21am

Hi @RoSchmi

This is a common issue at the moment where it stops at start_ssl_client function.

If you try this demo and output the debug log, the problem is the same as yours.

We are testing whether this problem is with the RTL SDK or with our communication protocol itself. If it is a problem with the RTL SDK, we will get in touch with Realtek to solve this issue.

Best Regards,
Lakshantha

RoSchmi · January 6, 2021, 9:07am

Hi @Iakshan,
I’m happy to see that you are searching the cause and I’m sure you’ll find it soon.
My investigations let me think that it might be a similar reason as posted here:

At least if I add a ‘_connected = false’ in the WiFiClientSecure::stop() routine, my application continues to work after a failed upload.
(However it may be that the true cause is somewhere else)
Best regards
RoSchmi

RoSchmi · January 8, 2021, 6:49pm

Good news !! Now with new RTL8720 firmware (seeed-ambd-firmware) v2.1.1 in a new test run until now 650 successful uploads (https) and still running. There was one failed upload (is expected to can happen) but the program continued to perform successive uploads as expected. Watchdog was not activated.

lakshan · January 8, 2021, 7:00pm

that’s great! This new firmware is meant to fix the SSL issues.

Hope you can keep running your demo

RoSchmi · January 11, 2021, 10:25am

Unfortunately I was happy too early. Stopping of uploads seems to occure now much less often but very rarely my App still stops uploading (all subsequent push request return with error code -1) while the loop is still running.
Does it work endlessly on your side with firmware v2.2.1?

lakshan · January 12, 2021, 12:35pm

Hi @RoSchmi,
Could you try changing the delay value here Seeed_Arduino_rpcUnified > src > erpc > erpc_arduino_uart_transport.cpp (line 299) on your end and test it? The delay seems to be the issue. When the delay is too high, it seems to be slow, and when the delay is too low, it seems to be unstable.

github.com

Seeed-Studio/Seeed_Arduino_rpcUnified/blob/master/src/erpc/erpc_arduino_uart_transport.cpp#L299


  return kErpcStatus_Success; // return size != bytesRead ? kErpcStatus_ReceiveFailed : kErpcStatus_Success;
}
erpc_status_t UartTransport::underlyingSend(const uint8_t *data, uint32_t size)
{
  uint32_t sentSize = 0;
  while (sentSize < size)
  {
    const uint32_t sendSize = min(size - sentSize, 32);
    sentSize += m_uartDrv->write(&data[sentSize], sendSize);
    delay(4);
  }
  return kErpcStatus_Success; // return size != offset ? kErpcStatus_SendFailed : kErpcStatus_Success;
}
bool UartTransport::hasMessage()
{
  if (m_uartDrv->available())
  {
    return true;
  }

Best Regards,
Lakshantha

RoSchmi · January 12, 2021, 4:51pm

Thanks, I’ll try, actually 5 ms delay.
BtW: Is there a way to reset the MCU -> RTL8720 connection without rebooting the complete board. So one could start a retry of the ssl transmission and would not loose data in ram.

lakshan · January 13, 2021, 3:17am

Hi @RoSchmi,
Please add the following codes where you need to restart the MCU -> RTL8720 connection.

 pinMode(RTL8720D_CHIP_PU, OUTPUT); 
 digitalWrite(RTL8720D_CHIP_PU, LOW); 
 delay(500); 
 digitalWrite(RTL8720D_CHIP_PU, HIGH);  
 delay(500);

Best Regards,
Lakshantha

RoSchmi · January 13, 2021, 10:02am

Thanks @lakshan . Unfortunately this code alone didn’t work to recover the connection. Probably it would also be needed to reinitialize the serial port of the SAMD51 MCU.
I think it would be too difficult to find out how this could be done.
Best Regards
RoSchmi

lakshan · January 13, 2021, 10:23am

Hi @RoSchmi,
You need to initialize the Wi-Fi libraries again after the codes that I have mentioned above.

Best Regards,
Lakshantha

RoSchmi · January 13, 2021, 11:09am

Thanks @lakshan, I’m not sure which code you mean with:

However, before trying this I would like to rule out that there is a memory leak causing the issue.
Do you have some routines which you use on the SAMD51 to test for memory leaks?

lakshan · January 13, 2021, 11:21am

@RoSchmi,
I mean these codes.

 pinMode(RTL8720D_CHIP_PU, OUTPUT); 
 digitalWrite(RTL8720D_CHIP_PU, LOW); 
 delay(500); 
 digitalWrite(RTL8720D_CHIP_PU, HIGH);  
 delay(500);

You need to reinitialize the Wi-Fi libraries after the above codes.

To be clear:

Include the above codes inside your demo, where you need to restart the MCU -> RTL8720 connection.
Reinitialize the Wi-Fi libraries

Will get back to you if there is any routine to use on the SAMD51 to test for memory leaks.

RoSchmi · January 13, 2021, 11:43am

I think, that I have already found some code;
Edit: The code I posted here before didn’t work for me as expected.
Now I’m using this code in my loop to check if there is a memory leak:
(depends on rpcWiFi library)
Edit2: Using pvPortMalloc() is not correct, malloc() has to be used, code is corrected

  // Keep track of tries to insert and check for memory leak
  insertCounterAnalogTable++;
uint32_t * ptr_one = (uint32_t *)malloc(100);
free(ptr_one);
if (insertCounterAnalogTable == 1)
{          
     referenceHeapAddr = (uint32_t)ptr_one;
}
lostLeakageBytes =  (uint32_t)ptr_one - referenceHeapAddr;
char buf[25] {0};
sprintf(buf, "Allocating at: %10X", (unsigned int)ptr_one);
Serial.println(buf);
sprintf(buf, " Lost %i",  lostLeakageBytes);
Serial.println(buf);

Result: There seems to be no memory leak

lakshan · January 14, 2021, 9:40am

that’s great. Thanks for sharing!

RoSchmi · January 14, 2021, 10:24am

Hi @lakshan,
I made tests with 5ms and 6ms delay. Didn’t solve the issue.
In my application it is not needed to store data in Ram permanently, so I can use the watchdog or reboot after a failed upload. With these measures I can do my uploads permanently. For other applications I will have to wait on your team to solve the problem.
Best Regards
RoSchmi

lakshan · January 15, 2021, 2:54am

Hi @RoSchmi,
Could you test with a higher delay such as 10ms or 20ms?

Great to hear that you have found a workaround for your application!

Can I know what other applications are you referring to?

Best Regards,
Lakshantha

RoSchmi · January 15, 2021, 10:20am

Hi @lakshan, thanks for your interest,

yes, I can test with 10 or 20 ms if needed.

Actually my App is running flawlessly (delay 4 ms) for 9 hours without using the watchdog, that’s a new record !! It’s a pity that I have to stop it now to continue programming.
Did you make any changes in the rpcUnified library update of yesterday which could have been a game changer?

I still have no other serious applications. If I had an idea of an internet using App which has the need of keeping data in the Ram I would not start before the platform has proved to run stable over longer periods of time.