System.OutOfMemoryException

[Matt.j5b]

I tried updating my CMX from 3.23.1 (b3221) to 3.25.2 (b3245) and get an Out of MemoryException and CMX crashing within 2 hrs of starting up.

This is using a Davis WLL on a Raspberry Pi 3B+. While running the newer version I notice the mono process eating through memory very quickly, and gets to below 50 MB many times, whilst a sizable swap disk has plenty of capacity. With v3.23.1 I don't get any issues using up memory quickly with free memory not falling below 200 MB.

Attached is Diags file with debug logging enabled.

Code: Select all

System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
  at (wrapper alloc) System.Object.AllocVector(intptr,intptr)
  at System.Text.StringBuilder..ctor (System.String value, System.Int32 startIndex, System.Int32 length, System.Int32 capacity) [0x00099] in <d636f104d58046fd9b195699bcb1a744>:0 
  at System.Text.StringBuilder..ctor (System.String value, System.Int32 capacity) [0x0000f] in <d636f104d58046fd9b195699bcb1a744>:0 
  at CumulusMX.WeatherStation.GetTempGraphData (System.Boolean incremental, System.Boolean local, System.Nullable`1[T] start) [0x0000d] in <7bc6b971a4244cf384fe58b0e2381f1f>:0 
  at CumulusMX.WeatherStation.CreateGraphDataJson (System.String filename, System.Boolean incremental) [0x002c9] in <7bc6b971a4244cf384fe58b0e2381f1f>:0 
  at CumulusMX.WeatherStation.CreateGraphDataFiles () [0x0004e] in <7bc6b971a4244cf384fe58b0e2381f1f>:0 
  at CumulusMX.Cumulus.DoHTMLFiles () [0x000f9] in <7bc6b971a4244cf384fe58b0e2381f1f>:0 
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__7_1 (System.Object state) [0x00000] in <d636f104d58046fd9b195699bcb1a744>:0 
  at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (System.Object state) [0x00007] in <d636f104d58046fd9b195699bcb1a744>:0 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <d636f104d58046fd9b195699bcb1a744>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <d636f104d58046fd9b195699bcb1a744>:0 
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00021] in <d636f104d58046fd9b195699bcb1a744>:0 
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <d636f104d58046fd9b195699bcb1a744>:0 
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <d636f104d58046fd9b195699bcb1a744>:0 
2023-07-24 08:20:00.472 DoLogFile: MySQL executing - INSERT IGNORE INTO Monthly (LogDateTime,Temp,Humidity,Dewpoint,Windspeed,Windgust,Windbearing,RainRate,TodayRainSoFar,Pressure,Raincounter,InsideTemp,InsideHumidity,LatestWindGust,WindChill,HeatIndex,UVindex,SolarRad,Evapotrans,AnnualEvapTran,ApparentTemp,MaxSolarRad,HrsSunShine,CurrWindBearing,RG11rain,RainSinceMidnight,WindbearingSym,CurrWindBearingSym,FeelsLike,Humidex) Values('2023-07-24 08:20',13.8,79,10.2,4,10,183,0.0,0.0,1021.9,476.0,17.4,57,4,13.8,13.8,0.0,104,0.05,641.52,13.1,310,0.0,205,0.0,0.0,'S','SSW',13.6,15.1)
2023-07-24 08:20:00.495 CustomSqlMins[0]: Error encountered during MySQL operation = Insufficient memory to continue the execution of the program.
2023-07-24 08:20:00.497 CustomSqlMins[0]: Error - Object reference not set to an instance of an object.
2023-07-24 08:20:00.643 DoLogFile: MySQL 1 rows were affected.
2023-07-24 08:20:00.967 FTP[196]: Error uploading /public_html/data/realtimeextras.txt : Error while uploading the file to the server. See InnerException for more info.
2023-07-24 08:20:00.967 FTP[196]: Inner Exception: Insufficient memory to continue the execution of the program.

Thanks!

[picca]

Maybe related?
viewtopic.php?t=21303

Do you have a "process kill message" in the system log as well?

For me the problem still occurs sporadically (but not many times a day, as it did before disabling swap on the raspberry). I use Monit to monitor and restart the process if killed.

[Matt.j5b]

Seem pretty similar.

Until I increased the swap size CMX would silently crash, aborted by the system like you say. Once I increased the swap size it work take longer to crash and cumulus would crash as well but not killed quite as often by the system.

Code: Select all

Jul 23 20:35:58 raspberrypi kernel: [ 3324.673775] Out of memory: Killed process 1295 (mono) total-vm:642028kB, anon-rss:520896kB, file-rss:0kB, shmem-rss:4kB, UID:0 pgtables:1018kB oom_score_adj:0

I didn't get this issue like this until upgrading to v3.25.2. Previously memory would get completely used up but would take much longer of which I use a script to restart the system when CMX is no longer running.

I have since reverted back to v 3.23.1 as this is happening every few hours, or less and is too unreliable.

[ConligWX]

just a thought.

what version of Mono are you running?

from Release 3.25.0 - build 3241 it requires Mono version 6.8 or later to run on Linux

[Matt.j5b]

I am running Mono version: 6.12.0.200.

[freddie]

Are you using the new php upload functionality?

[Matt.j5b]

No I am not using the PHP upload, and have continued to use FTPS.

[picca]

If it helps... same for me. As I wrote in the other thread, no php, only mqtt, mysql and ftps.

At this point I think the version of mono (still 6.8 for me) doesn't matter.

[f4phlyer]

I also have had the ‘memory exception’ since 3235 till now. Thanks to Mark I have been running a beta version of 3248 for over a week and would like to report that it has been running flawlessly.

I would attribute the ‘fix’ to the cumulative changes to MX (rewrites, recompiles etc.), updates to the OS (Raspian) and Mono I’ve been running the beta for over a week with no indication of a ‘memory leak’. I have 3 instances of of MX running on various Raspberry pi’s.

So if you’ve been holding off on trying a new version you might try CumulusMXDist3248.