CustomLogs as module

[HansR]

Hi,

Currently, Cutils analyzes all files that are defined in the Custom Logs tab. I have 2 files there, one is drawn by the module and the other is not. Additionally, the second file contains prohibited webatags, i.e. dates, and I don't want to draw them.
The problem is that this second file with dates creates large reports. Additionally, it slows down the computer significantly.
Is it possible and sensible to create a parameter where it will be possible to define a custom log to be taken into account by Cutils?

For hypothetical example in cumulusutils.ini

Code: Select all

[General]
CustomLogFile=customlog.txt -- here is the name of the file that would be executed. If empty, it would remain as it is now.

In this case, Cutils will only deal with the specified file, not all of them.

Hi,

I have some questions:

1. Does the customlog function correctly i.e. it only plots the plottable webtags? Dates should not be plotted by definition.
2. What does the computer slow down?
3. Could you split up the table (in 2 or more), have them plotted and make a plot with the webtags you wish?
4. Are you using the cutomlog for other things than plots, Do you have an example in your website?
5. Or create a customlog with only the valid webtags? What do you need the dates for?
6. Could you post the definition of the customlog as found in Cumulus.ini to give me an indication of what we are talking about? I may have other ideas.

To answer your basic question: yes I assume that is possible. But I would like to understand the why.

[Dador]

1. The file is intended for creating a chart, it draws well. I have no problems with it.
2. Run cumulusutils.exe with the CustomLogs argument.
3. I only have a problem with the file that I don't want to draw. There are dates in it, but I won't remove them.
4. Customlogs by Cutils is only for drawing defined charts.
5. Dates are necessary in a file that I need for purposes other than drawing charts.
6. I'm creating something like this.

Code: Select all

[CustomLogs]
DailyEnabled0=1
DailyFilename0=dayfile_cvp
DailyContent0=<#wgustTM rc=y>,<#bearingTM rc=y>,<#TwgustTM>,<#tempTL rc=y>,<#TtempTL>,<#tempTH rc=y>,<#TtempTH>,<#pressTL rc=y>,<#TpressTL>,<#pressTH rc=y>,<#TpressTH>,<#rrateTM rc=y>,<#TrrateTM>,<#rfall rc=y>,<#avgtemp rc=y>,<#windrun rc=y>,<#windTM rc=y>,<#TwindTM>,<#humTL rc=y>,<#ThumTL>,<#humTH rc=y>,<#ThumTH>,<#ET rc=y>,<#SunshineHours rc=y>,<#heatindexTH rc=y>,<#TheatindexTH>,<#apptempTH rc=y>,<#TapptempTH>,<#apptempTL rc=y>,<#TapptempTL>,<#hourlyrainTH rc=y>,<#ThourlyrainTH>,<#wchillTL rc=y>,<#TwchillTL>,<#dewpointTH rc=y>,<#TdewpointTH>,<#dewpointTL rc=y>,<#TdewpointTL>,<#domwindbearing rc=y>,<#heatdegdays rc=y>,<#cooldegdays rc=y>,<#solarTH rc=y>,<#TsolarTH>,<#UVTH rc=y>,<#TUVTH>,<#feelslikeTH rc=y>,<#TfeelslikeTH>,<#feelslikeTL rc=y>,<#TfeelslikeTL>,<#humidexTH rc=y>,<#ThumidexTH>,<#chillhours rc=y>,<#rain24hourTH rc=y>,<#Train24hourTH>
IntervalEnabled0=1
IntervalFilename0=customlog
IntervalContent0=<#temp rc=y>,<#hum rc=y>,<#dew rc=y>,<#wetbulb rc=y>,<#wchill rc=y>,<#heatindex rc=y>,<#humidex rc=y>,<#apptemp rc=y>,<#feelslike rc=y>,<#avgtemp rc=y>,<#THWindex rc=y>,<#THSWindex rc=y>,<#ExtraTemp1 rc=y>,<#ExtraHum1 rc=y>,<#ExtraDP1 rc=y>,<#ExtraTemp3 rc=y>,<#ExtraTemp4 rc=y>,<#ExtraHum4 rc=y>,<#ExtraDP4 rc=y>,<#SoilTemp1 rc=y>,<#SoilTemp2 rc=y>,<#SoilTemp3 rc=y>,<#SoilTemp4 rc=y>,<#SolarRad rc=y>,<#cloudbasevalue rc=y>,<#rfall rc=y>,<#rrate rc=y>,<#rhour rc=y>,<#r24hour rc=y>,<#rmonth rc=y>,<#ryear rc=y>,<#StormRain rc=y>,<#LightningDistance rc=y>,<#LightningStrikesToday rc=y>,<#AnnualET rc=y>,<#SunshineHoursYear rc=y>,<#UserTemp1 rc=y>,<#wspeed rc=y>,<#wgust rc=y>,<#avgbearing rc=y>,<#press rc=y>,<#altimeterpressure rc=y>,<#stationpressure rc=y>
IntervalIdx0=4

[HansR]

I don't understand what you are doing but I'll make a trick to exclude a customlog in CUtils.

[Dador]

I don't understand what you are doing but I'll make a trick to exclude a customlog in CUtils.

If you want, take a look at the effects of CustomLogs' work.

https://pogodarybnik.pl/CU/customlogsRecentdata.json (file weighs ~71 MB!)

https://pogodarybnik.pl/CU/customlogsDailydata.json (file weighs ~70 MB!)

Exemplary utilslog

https://pogodarybnik.pl/CU/2405061125cumulusutils.log (file weighs ~11 MB!)

Maybe you can understand me, why I don't want Cutils to deal with the dayfile_cvp file.

[HansR]

I don't understand what you are doing but I'll make a trick to exclude a customlog in CUtils.

If you want, take a look at the effects of CustomLogs' work.

https://pogodarybnik.pl/CU/customlogsRecentdata.json (file weighs ~71 MB!)

https://pogodarybnik.pl/CU/customlogsDailydata.json (file weighs ~70 MB!)

Exemplary utilslog

https://pogodarybnik.pl/CU/2405061125cumulusutils.log (file weighs ~11 MB!)

Maybe you can understand me, why I don't want Cutils to deal with the dayfile_cvp file.

Hi,

I don't know what is happening but there is seriously something wrong with your data design: in your Recentdata datalog there is for each parameter (e.g. customlogtemp as first parameter) 43 value pairs as "[[1714503000000,25.3] i.e. on the same time the same value.

I don't know about the how and what but this is not as it should be. If this is a minute value logging why does it do the same time/value combination 43 time? That is one way to fill up your database!

I would say check your definition of the customlogs, 43 times the same time/value combination is not correct. It may be a bug in CUtils but with this amount of data I can't analyse this. How many datalogs are used to create this json?

Do you maybe duplicate those webtags in more than one customlog... it is a riddle to me.
Do you have temperature in 43 DAILY customlogs perhaps?
NOTE: all customlogs on the DAILY frequency get their data in the RECENT.json sent to the server. It seems you are just overcharging the system: all data are in ascii in the json, that gives an issue on size. You have to calculate what you can handle.

What do you want to do with these customlogs (data)? Do you want to create a report with the dates? Currently you make customlogs and you want to ask the only tool that is able to process customlogs to quit processing... I have difficulty following the reasoning (though I understand you have a problem )

Yes I can skip a customlog in CUtils - I already coded it - but I still don't see the use of it, certainly not when looking at this kind of datafiles.
My primary guess is you have an error in customlog design.

[Dador]

I've been using CustomLogs for about half a year now. I had no problems before. It all started yesterday after updating to CMXv4 and Cutils 7.0.0.

I haven't changed anything in the settings since then. Everything is as it was.

I create Custom Dayly Logs , but for purposes other than drawing in Cutils.

Because what if I want many customlog files with different tags? Cutils will analyze all of them, but for example I want it to analyze only one of these files. That's all. As always, I want to have a choice. Sorry, but I can't explain it any simpler.

[HansR]

That is strange... There is no change in the logic between v3 and v4.

I think we have a similar sensor config (though i have no c02) i could try test setup and see what happens. Meanwhile you could check your setup. This way we can try to find out what is wrong.

Could you PM me the Cumulus.ini and the cumulusutils.ini and i'll see what i can do.

[Dador]

Ok, but to test my situation you need to create a file with time tags. You can use the tags I provided above and I use them myself.
I think it's all because of these time tags.
However, I would like to point it out right away. I won't delete them because I need them in another script.
For now, I have disabled CustomLogs logging. This generated quite a lot of network traffic.

[HansR]

Ok, but to test my situation you need to create a file with time tags. You can use the tags I provided above and I use them myself.
I think it's all because of these time tags.
However, I would like to point it out right away. I won't delete them because I need them in another script.
For now, I have disabled CustomLogs logging. This generated quite a lot of network traffic.

Find attached a test version 7.0.1 (see release note) which should fulfil your requirement. Just fill in ExcludedCustomLogs with a comma separated list of customlogs you do not wish CUtils to process.

Let me know how you get on and I am curious about the result of your other procedure (report?) using the customlogs.

But as far as I can see your database fills up with 43 equal data pairs for each entry and for each webtag. That is - must be - an error somewhere.
Could you post the customlog datafile (the interval customlog customlog, maybe you should consider giving it a meaningful name).

So that should be something like customlog-202405.txt .
I would like to check the data vs the json.

[HansR]

OK. I may have found an issue which generates the data 43 times, with 43 being the number of tags in the CustomLog.
I will examine further... and keep you posted.

[Dador]

The latest feature works properly

Code: Select all

[CustomLogs]
ExcludedCustomLogs=

In fact, I still have a problem with customlogsRecentdata.json
I look at the data in the /data folder, but it all seems correct

[HansR]

Yes, there was an issue in the JSON creation so if you do not exclude your CustomLogs you will still see the massive JSON files.
I fixed it and created a second test version for you to try. When generating the JSON it should be 1/43 of the size you saw before (in your case). That should significantly improve your performance.

The exclusion functionality will remain since CustomLogs are basically a CMX functionality and it indeed should be possible to exclude this from CUtils on a per CustomLog basis (you could already exclude the module as a whole).

If you fill in the CustomLog name(s) in the ExcludedCustomLogs those CustomLogs will be omitted from CUtils and no JSON for those logs will be created. In your case: fill in the two names (comma separated list).

Let me know how this test 2 functions, please send me your resulting JSON (or link to it) again.

[Dador]

Thank you, it helped.

https://pogodarybnik.pl/CU/customlogsRecentdata.json (file weighs ~1,6 MB)

[HansR]

Thank you, it helped.

https://pogodarybnik.pl/CU/customlogsRecentdata.json (file weighs ~1,6 MB)

OK. That looks better.

Is the result of your script also on the website? If so, where?
And do you keep the charts now? If so... where?

[Dador]

Is the result of your script also on the website? If so, where?
And do you keep the charts now? If so... where?

Here are the charts generated by CustomLogs.

https://pogodarybnik.pl/customcharts.htm

Edit:

Request to unblock webtag

<#TempAvg24Hrs>