Chart display problems and data download.

[PaulMy]

Hi,
That appears to be the same as pointed our earlier. You need to again open dayfile.txt and look at that line 912 (and also all the lines to 1011). Check each data item to ensure it is in a valid form, i.e. numbers or date, and commas are used to separate data items, and periods used for number with decimal places. To find the previous error I copied a couple of lines above the error line into a separate Notepad++ page and then also to line in question (i.e. 912). Then using tabs to line up the data in each of the lines, and that helps to find the error. In my earlier post I mentioned your pressure with a comma decimal, but it is the same for many of the other data items. Your dayfile.txt lines up to 911 seem to be okay (I have not checked them all and only using the error lines from your MXdiags). The lines from 912 to 1009 contain mostly comma decimals. From lines 1011 it is back to period decimals. Possibly other lines can also have the issue. If so, CumulusMX should indicate the error in MXdiags. Here is my worksheet that clearly shows commas instead of periods:

Code: Select all

Line No.Date (dd/mm/yy)	Max gust	Max gust bearing	Max gust time	Min temp	Min temp time	Max temp	Max temp time	Min pressure	Min pressure time	Max pressure	Max pressure time	Max rainfall rate	Max rainfall rate time	Total rainfall	Avg temp	Total wind run	High avg wind speed	High avg wind speed time	Low humidity	Low humidity time	High humidity	High humidity time	Total ET	Total hours of sunshine	High heat index	High heat index time	High apparent temp	High apparent temp time	Low apparent temp	Low apparent temp time	High hourly rain	High hourly rain time	Low wind chill	Low wind chill time	High dew point	High dew point time	Low dew point	Low dew point time	Dominant wind bearing	Heating degree days	Cooling degree days	High solar rad	High solar rad time	High UV-I	High UV-I time	High feels like	High feels like time	Low feels like	Low feels like time	High humidex	High humidex time	Chill hours	High 24 hour rain	High 24 hour rain time
910		27/09/17,		22.0,		180,				09:30,			12.9,		00:00,			16.4,		18:30,			1011.01,		17:50,				1018.73,		00:00,				1.36,				11:40,					1.18,			14.4,		95.3,			8.0,				08:20,						95,				02:10,				99,				18:10,				0.000,		0.0,					16.4,			18:30,					17.4,				19:00,					12.7,				00:00,					0.30,				10:10,					12.9,			00:00,				16.3,			18:30,				12.1,			03:00,				145,					3.9,				0.0,				0,				00:00,				0.0,		00:00
911		28/09/17,		20.0,		180,				17:50,			8.6,		06:10,			17.3,		16:00,			1013.92,		19:50,				1018.56,		08:40,				0.26,				18:50,					0.13,			14.1,		60.7,			6.0,				13:10,						87,				14:30,				100,			07:50,				0.000,		0.0,					17.3,			16:00,					17.4,				21:40,					8.2,						06:10,					0.09,				19:00,					8.6,			06:10,				16.2,			20:50,				8.4,			06:10,				166,					4.2,				0.0,				0,				00:00,				0.0,		0:00
912		29/09/17,		15,0,		180,				01:50,			11,4,		07:50,			17,2,		15:00,			1013,24,		01:30,				1018,05,		20:50,				0,44,				03:10,					0,00,			14,4,		33,2,			5,0,				00:50,						62,				15:10,				99,				08:20,				0,000,		0,0,					17,2,			15:00,					16,7,				14:50,					11,7,				07:50,					0,32,				08:58,					11,4,			07:50,				15,6,			01:00,				9,8,			15:20,				197,					3,9,				0,0,				0,				00:00,				0,0,		00:00
913		30/09/17,		20.0,		135,				15:30,			12,2,		04:20,			16,5,		11:30,			1009,79,		18:50,				1017,37,		00:00,				0,42,				13:50,					0,69,			14,5,		65,7,			7,0,				14:40,						86,				00:40,				99,				19:10,				0,000,		0,0,					16,5,			11:30,					16,8,				19:10,					12,4,				03:30,					0,21,				14:30,					12,2,			04:20,				15,5,			19:10,				11,0,			03:30,				172,					3,8,				0,0,				0,				00:00,				0,0,		00:00
914		01/10/17,		14.0,		225,				14:50,			14,2,		03:40,			17,4,		14:40,			1010,36,		09:40,				1015,27,		00:00,				0,14,				08:50,					0,21,			15,8,		58,3,			5,0,				12:20,						91,				16:50,				99,				00:00,				0,000,		0,0,					17,4,			14:40,					19,2,				14:40,					14,8,				04:50,					0,06,				8:50,					14,2,			03:40,				17,0,			13:30,				14,0,			03:40,				196,					2,5,				0,0,				0,				00:00,				0,0,		00:00

1011		06/01/18,		15.0,		135,				17:20,			-1.0,		07:40,			7.6,		12:50,			1000.10,		00:00,				1019.74,		23:40,				0.00,				00:00,					3.01,			3.0,		45.3,			6.0,				15:50,						82,				14:00,				99,				09:10,				0.000,		0.0,					7.6,			12:50,					5.9,				12:50					-3.2,				07:40,					4.85,				10:40,					-1.0,			07:40,				6.0,			12:20,				-1.3,			07:40,				132,					15.3,				0.0,				0,				00:00,				0.0,		00:00

Enjoy,
Paul

[sfws]

PaulMy is doing an excellent job of helping you, and I'm not about to take over.

However, Paul did say he was not familiar with technical issues, so I don't think he will mind if I mention what is causing your issues, and I expect you will find the background explanation removes some of your frustration. Don't be put off by length of this reply, I'm trying to ensure I have covered everything!

The following relevant lines appear in 20221014-141554.txt, one of the MXdiags files you zipped up earlier:

Code: Select all

2022-10-14 14:15:56.195 Directory separator=[\] Decimal separator=[.] List separator=[,]
2022-10-14 14:15:56.196 Date separator=[/] Time separator=[:]

You are using Microsoft Windows, in the Control Panel of that there is a section called Region.
Select that, and a tab labelled Formats appears, press a button there labelled Additional settings... and a Customise Format window opens.
The default tab for that is headed Numbers, that is where the cumulus software is being told what your decimal symbol is and what your list separator is.

You have got the problem because those lines were created during a period when your computer that you were using (for some reason) had decimal point and list separator settings both set to comma (rather stupidly) for those days. Therefore in any line that is wrong, every field that expects to see a decimal point on that line will be missing that decimal point. and it is hard to work out which commas are decimal separators and which are list separators! Look at field list documentation to double check which fields must not have a decimal.
You are getting the error message whenever Cumulus first finds a line that it cannot understand based on these settings, basically it assumes every time it finds a comma that it has moved onto a new field, but sometimes that new field contains non-comma characters it cannot match to what it expects to find in that field.

Should you mess up the editing remember there is a backup copy of the file, in state as it was after last rollover, stored in the backup\daily directory, in the sub-directory with the most recent date/time.

(POST EDIT: REPHRASED SOME OF WHAT FOLLOWS)
Don't expect the error message to pinpoint first error. The MX software attempts to read the file into an array (without line ending characters) held in RAM, and when a line cannot be understood, there can be errors re deciding where line starts/ends as well as deciding what field goes in what position. (The MX coding regarding dayfile.txt array handling was altered in a fairly recent release). Therefore the actual errors may be in slightly earlier lines or in different positions in the line. Subsequent errors in that line or other lines may not be reported.

As Paul has also advised, because Cumulus does not recognise the lines, you need to edit the file outside the Cumulus software. You can leave Cumulus running while you edit the file. You mention that you are using Notepad++, that is a good choice, (although there are other code editing programs that will also run on your PC).

Basically, there are six tasks to do with your Notepad++ (or another tool):
First, open "dayfile.txt"
Second, Notepad++ (and other good tools) will number the lines, so you can quickly navigate to a good line a couple of lines before the line number mentioned in the error message (you want to match the pattern seen in a good line of where decimal points should be included)
Third, insert (on a temporary basis, it later has to be removed), the contents of dayfileheader.txt just before that good line (so that you align the times in that good line with the time fields in the labels, having times aligned makes it easier to spot the value fields that are placed between time fields)
Fourth, space out the next few lines (these are lines where errors might be), so you can inspect all the punctuation to see if it matches what cumulus expects (a decimal point, not a decimal comma within fields and a comma between fields), and change it where it does not. I would be more fussy than in Paul's worksheet to get alignment same between the titles taken from dayfileheader.txt and the fields in lines taken from dayfile.txt!
(You may need to move the line with labels further down, as you continue with subsequent bad lines, repeating step above)

Fifth, get all lines in file back to how Cumulus expects them (no extra spaces, no blank lines), and no field title lines
Finally, save file back into the data directory in an encoding format that Cumulus can understand. Don't worry too much about this, most tools like Notepad++ will not change the original format.
(For Microsoft Notepad, and Google editors, the default encoding generally adds an unwanted Byte Order Mark (UTF-8 with BOM), so for them you need to choose Save as, then choose Encoding setting of UTF-8 on that "Save as" screen).

Once you have successfully edited, and saved the file back into Cumulus, use the Utils menu Reload dayfile. Remember that Cumulus works from a copy of the file held in RAM, so without this extra step the software will be unaware the file has changed.

[JohnF]

Thank you both for your excellent replies and I shall implement your suggestions after a, hopefully, good night's sleep
I do appreciate your help and time over my problems.
Cheers
John

[JohnF]

Update. There are 100s of errors in the dayfile which I am gradually working through.
Some lines have only one error and others 2 or 3.
I suspect a Windows update caused the change from . to ,
John

[JohnF]

I think I have corrected all of the errors so would you please check the MXdiags folder for me please?
I have only one issue. When I run the localhost web Firefox does not respond.
After closing Firefox, the localhost web will then run OK.
Thank you again guys.

[PaulMy]

Hi,
The first 7 MXdiags show the issue in dayfile, but the last 5 are ok, so looks like you fixed up the data
I can't detect anything else in the files so don't know what the Firefox issue may be. Is it the same if you connect with localhost:8998 and {yourIPnbr}:8998 Have you tried another browser?

Enjoy,
Paul

[JohnF]

Hi Paul.
Thank you for checking and confirming what I thought.
It is Firefox that sometimes stops responding and Edge seems OK.
I can live with that.
Many, many thanks for your time and help.
John

[PaulMy]

You're welcome. Glad it is working well again.

Enjoy,
Paul