[canemxiao]

Hi all,

It seems that Knossos Explore tab is acting up on me and not displaying any mods.
Knossos log:

Code: [Select]

INFO:Thread-7:util.download:Downloading "https://cf.fsnebula.org/storage/repo.json"...
ERROR:Thread-7:tasks.work1: Failed to decode "#public"!
Traceback (most recent call last):
  File "C:\knossos\knossos\tasks.py", line 107, in work1
  File "C:\knossos\knossos\repo.py", line 128, in parse
  File "C:\Python36\lib\json\__init__.py", line 354, in loads
  File "C:\Python36\lib\json\decoder.py", line 339, in decode
  File "C:\Python36\lib\json\decoder.py", line 355, in raw_decode
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1244785 (char 1244784)

For whatever reason, it is not recognising repo.json as a valid json

I was using Knossos for quite a while now but recently it broke down, could you guys help take a look?

Many thanks

[canemxiao]

Ok... I played around with Knossos and I think I've found the issue.

The reason seems to be downloading of https://cf.fsnebula.org/storage/repo.json stopped midway leaving a partially formed mods.json under C:\Users\<username>\AppData\Roaming\knossos which Knossos failed to read.
I downloaded the JSON using a downloader and replaced mods.json with the downloaded file, which fixed the issue.

however, that partially formed JSON is already 116MB+ and I suspect that due to python being slow as all hell, it never finished downloading before the timeout and then the connection closed.
the full JSON is around 270MB which brings me to the questions:

1) why does have to be this big?
2) could we do some optimisation on the size or how it downloads mod list?

[jg18]

Hi there! Thanks for finding this. The JSON file probably used to be a reasonable size but kept growing as the number of mods listed on Knossos grew.

There are a variety of options for a solution, such as providing only limited info about each mod and then providing details on demand, but the result may be chatty client-server I/O. Another option is versioning the JSON and sending delta updates from the server, rather than the full JSON file every time. Other solutions undoubtedly also exist.

Unfortunately I do not have bandwidth to look into this but perhaps someone else does. You may want to file a GitHub issue (the repo is ngld/old-knossos IIRC).

The workaround you found may be the best option for now.

EDIT: an even simpler code fix is to send zipped JSON and unzip client-side. The JSON should compress well, being so text heavy and likely with similar language throughout. I bet bzip2 would work well, but not sure offhand if Python supports that.

EDIT 2: .7z would be better than bzip2. Knossos already uses it for mod data.

[Erkhyan]

I’ve recently gotten into FS2 again and immediately ran into that huge JSON file issue. FYI, I randomly checked, and a 7z-compressed version of the file would be roughly 10mb, versus how impractically large it is now.

[Cyborg17]

We've had reports of this issue on discord already, and the stopgap to fix this issue until we get knossos to deal with it better is to delete %appdata%/roaming/knossos/mods.json and %appdata%/roaming/knossos/mods.json.etag

This forces knossos to retry the download. 

Taylor has been doing some work on knossos lately, but he's not currently able to fix this issue.


The cause for this issue is that the download sometimes fails without knossos realizing it.