JSON.decode is so slow to be unuseable for any complex JSON#968

Parsing the JSON returned by this web request:
https://api.scryfall.com/cards/search?unique=prints&order=released&q=name%3AHypnotic%20Specter

takes about 4 seconds (during which the UI gets stuck).
Please provide a more efficient implementation of JSON decoding. This would allow for massive possibilities of integration of TTS with the Internet.

JSON is used more and more in TTS, and the slow speed of the parser is becoming a bottleneck affecting more and more users.

I was able to write a Lua-based parser that’s 4x faster than the current one, but a C# implementation could be 100x faster.

MoonSharp exposes a JSON parser (json.parse) that is 100x faster than the one provided by TTS (JSON.decode), but it’s quite buggy. Speed is irrelevant if it doesn’t work!

In addition to using a C# implementation, a further improvement would be to allow Lua code to be run on a separate thread. For example, like
Unity’s Job System: https://docs.unity3d.com/Manual/JobSystem.html

This would allow long-running code to be run on a separate thread than the UI, so that it doesn’t block the UI.

In addition to using a C# implementation, a further improvement would be to allow Lua code to be run on a separate thread.That would be highly undersireable.

The whole point of having Lua code is to be able to manipulate the game state in response to events. Allowing the game state to change would defy the point.

There is already an asynchronous JSON encoder out there if you want one. Howvever, it’s not very useful since the reason the slowless of the existing one causes problem is that it’s often used in onSave to generate the value returned. So something asynchronous is useless in that situation.

The whole point of having Lua code is to be able to manipulate the game state in response to events. Allowing the game state to change would defy the point.

That’s a good point - most Lua code needs the game state to be frozen.

However, there are at least two scenarios where asynchronous code would be desireable:

1) To decode a large JSON web request, as asked for in the original post. As long as code is written defensively, it would be okay for the game state to continue changing while the JSON request is decoded.

2) During the onLoad() function of a Memory Bag or Utility Memory Bag. I have three mods (1, 2, 3) that freeze for about 5 seconds after the mod finishes loading because the mods contain several memory bags, each of which decodes a large JSON string during their onLoad() method. I would love to asynchronously decode the JSON so the mod doesn’t freeze after loading.

I will also point out, Tabletop Simulator already provides a Wait class for pseudo-asynchronous code. This further proves that there are use cases for asynchronous code that does not need the game state to be frozen.

This further proves that there are use cases for asynchronous code that does not need the game state to be frozen.

Noone said otherwise. If you’re now saying you want the ability to write asynchronous code, well you’ve just proven you already can. I even uses Promises (just like JS’s) in my TTS Lua modules.

I would love to asynchronously decode the JSON so the mod doesn’t freeze after loading.

Again, I’ve already stated this is not only already possible to write today, someone’s already done so. That those bags don’t use it is a matter you should take up with the author of those mods.

If you’re now saying you want the ability to write asynchronous code, well you’ve just proven you already can.

I apologize for not being clear. When I said “asynchronous” I meant “asynchronous and off the main thread”.

The key is “off the main thread” so the UI doesn’t get blocked.

There is already an asynchronous JSON encoder out there if you want one.

Thank you! This is great! Would you mind sharing where I can find it?

That those bags don’t use it is a matter you should take up with the author of those mods.

Luckily the Utility Memory Bag has a GitHub repo and takes pull requests. I’ll gladly make a pull request to make the onLoad() optionally asynchronous.

Bugs of json.parse of which I am aware:
https://discord.com/channels/342471570955960324/368208369812504587/1457432922603913448
(Links to a post in the #scripting channel of the official TTS Discord.)
I suspect all of these will be fixed by the aforementioned upgrade.

@kinithin aka ikegami the negative number issue (the most prominent issue people have had) has been fixed, but i’ve only just learned of the latter two issues, which are not yet fixed. i’ll look into it, will likely be fixed in the hotfix after the upcoming beta

thank you for reporting the extra issues!

as for the “parity” i mentioned:
MoonSharp’s implementation of the JSON serializer/deserializer is.. Not The Best right now.

• it handles functions, userdata, and coroutines poorly (JSON will throw an error, json will silently ignore them. no i am not kidding you. it literally just decides they dont exist and passes over them)
• it only permits tables (arrays/objects), while JSON permits any type (technically the JSON spec only allows document roots to be arrays/objects, but since the JSON library has established this precedent, we aren’t switching until json matches its behavior)

JSON will continue to be the same slow (written in lua) library until we have worked on the built-in json library enough to be comfortable with switching it

oh and also, JSON encodes an empty table as an empty array, json encodes it as an empty object

Just tried the beta. The new Moonsharp version only fixes -1, but not \u escapes and \/

the negative numbers patch is available on public_beta, other fixes are still in the works

Another bug in json.parse, it fails to parse arrays where the first element is an array if there is no white space between the left brackets. Ex: json.parse("[[1,2,3]]") fails but json.parse("[ [1,2,3]]") succeeds.

I’d make a separate request for that being quite independent imho